Cloudflare Developers•3y ago

Can I add data to a d1 database from workers and a python script?

Samantas5855OP•9/30/23, 10:04 PM

also does delete count as a write or is it free?

SSamantas5855 Can I add data to a d1 database from workers and a python script?

Cyb3r-Jak3•9/30/23, 10:19 PM

You'd would have the use the API to add the content from a python script

Cyb3r-Jak3•9/30/23, 10:22 PM

And I'd imagine that deleting rows counts as a write but not about if you delete a D1 database entirely

ajgeiss0702•10/1/23, 12:58 AM

Once d1 has read replication, any idea on how consistent it would be? I’m currently planning a feature that would require semi-strong consistency
(Please ping me if you reply)

Aakira.eth One thing I missed is the dynamic creation of databases. Every user that creates...

Matt Silverlock•10/1/23, 5:31 AM

d1-database

Aajgeiss0702 Once d1 has read replication, any idea on how consistent it would be? I’m curren...

Unsmart•10/1/23, 5:32 AM

What does "semi-strong consistency" mean its either strongly consistent or its not

I'd assume read replication would be eventually consistent

Sameer Ali•10/1/23, 7:46 AM

You can share .wrangler folder between multiple workers and point to same migration directory from your wrangler.toml

UUnsmart What does "semi-strong consistency" mean its either strongly consistent or its n...

ajgeiss0702•10/1/23, 12:19 PM

By semi-strong I meant under 5 seconds

UUnsmart What does "semi-strong consistency" mean its either strongly consistent or its n...

Matt Silverlock•10/1/23, 2:31 PM

Both terms are very binary. D1 will have causal (session) consistency - “read your own writes” - with read replication

UUnsmart What does "semi-strong consistency" mean its either strongly consistent or its n...

Matt Silverlock•10/1/23, 2:32 PM

https://jepsen.io/consistency

tadeumaia•10/1/23, 2:39 PM

With the release of hyperdrive, does it mean that today I could have a "read replica" of my data at all Cloudflare point of presence at my country, while with D1 I'll have to wait for the release of the read replicas? (being my country is Brazil, i don't even see a hint of D1 for south america)

Ttadeumaia With the release of hyperdrive, does it mean that today I could have a "read rep...

Matt Silverlock•10/1/23, 5:43 PM

Hyperdrive doesn’t create replicas: it caches reads. Big difference.

MMatt Silverlock Hyperdrive doesn’t create replicas: it caches reads. Big difference.

tadeumaia•10/1/23, 5:59 PM

I tried to imply that with the quotes, sorry for the miscommunication, and thanks for your answer. I’m looking for a pure performance standpoint. Looking to rewrite some parts of a product that I have for the maximum read performance. Right now all located on AWS SA and cloudflare on front caching the html static. Users all over Brazil, would love to have the data distributed for when the cache bursts. But I was actually reading hyperdrive doc further and the cache looks short for my use case as I don’t have that much reads because all is cached statically.

Ttadeumaia I tried to imply that with the quotes, sorry for the miscommunication, and thank...

Matt Silverlock•10/1/23, 6:01 PM

Got it. Hyperdrive is still going to be a very effective intermediate cache - on top of the connection pooling & perf. Without it, cache misses will likely hurt a lot in terms of latency.

Ttadeumaia With the release of hyperdrive, does it mean that today I could have a "read rep...

Matt Silverlock•10/1/23, 6:02 PM

wrt D1 in South America - we’re working on it.

MMatt Silverlock wrt D1 in South America - we’re working on it.

tadeumaia•10/1/23, 6:16 PM

Thanks. That’s great to know. Rewriting the stack from .net 4 to workers after the TCP connection announcement. Most of it is ready, was going with a hosted AWS Postgres for development but looking for the feasibility at the hyperdrive/D1 after this week’s announcement. Will hyperdrive work with D1 or will D1 have that builtin with replicas at all the pop’s?

Ttadeumaia Thanks. That’s great to know. Rewriting the stack from .net 4 to workers after t...

Matt Silverlock•10/1/23, 6:37 PM

Separate things. Hyperdrive for databases not on Cloudflare. D1 as D1. Search this channel for my posts on “read replication” for D1

mukhtharcm•10/2/23, 1:54 AM

is downloading the database coming back soon?

Mmukhtharcm is downloading the database coming back soon?

mukhtharcm•10/2/23, 1:54 AM

We had this ability to download backup, but since time machine came, we can't

Mmukhtharcm We had this ability to download backup, but since time machine came, we can't

Dahoom•10/2/23, 4:03 AM

https://github.com/nora-soderlund/cloudflare-d1-backups

GitHub

GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a ...

A package to dump a copy of your experimental D1 database through a worker. - GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a copy of your experimental D1 database through a worker.

DDahoom https://github.com/nora-soderlund/cloudflare-d1-backups

mukhtharcm•10/2/23, 4:17 AM

Ty!

Mmukhtharcm Ty!

Dahoom•10/2/23, 4:18 AM

dont thank me, thank nora. Shes the one who made it

Original message was deleted

mukhtharcm•10/2/23, 6:15 AM

Thank you @nora for this!
I've been searching forever for something like this!

AAlexander I'm guessing you have a production database of some kind and are you planning to...

Laurent Pellegrino•10/2/23, 12:54 PM

Thanks for your message.

Yep, it's somewhat like a read-only replica. But the challenge is that about 90% of the data is refreshed daily, so diff computing or event-based reactions won't be of much help. I think @rozenmd's generic response might be due to Cloudflare not wanting backups to be abused. If I'm not mistaken, backup imports don't count as writes. So, in theory, someone could set up a local database, fill it up, back it up, and then import to Cloudflare D1 at no cost. Those backups weren't just for disaster recovery; they fit my scenario perfectly. But with only Timetravel remaining, the $1/M write cost is steep for those of us needing to import large databases daily or even weekly only.

Max (@rozenmd)•10/2/23, 12:59 PM

backup imports don't count as writes

D1's beta backend doesn't have backup imports - you can write to the DB, and you're billed as a write.

Metrics do not appear for D1's older alpha dbs because the metrics feature was built only for the beta backend.

JJason Hostetter Great to hear, looking forward to it. Was worried that functionality would be lo...

Matt Silverlock•10/2/23, 1:40 PM

FYI: you will be able to create up to 50,000 D1 databases per account as of this week (and if you genuinely get close, we can increase further: this is a soft limit).

Steved•10/2/23, 1:55 PM

Great to hear. Will there be a way for a pages app, for example, to access the multiple databases without adding new bindings, and having to redeploy? Classic case of multi-tenancy. In other systems, you'd just load the DB connection string for the tenant then access the DB.

MMatt Silverlock FYI: you will be able to create up to 50,000 D1 databases per account as of this...

Altrius•10/2/23, 3:29 PM

heck, thats a lot of databases haha, if anyone manages to actually reach that ill be impressed (unless they do one db per user? which could make sense for some use cases I guess haha)

Epailes•10/2/23, 5:45 PM

"By default, D1 will automatically create your database in a location close to where you issued the request to create a database"
I hadn't even considered that instances will be close to users by default because of this, D1 will honestly be game changing for multi-tenant applications!

Epailes•10/2/23, 5:46 PM

I'm thinking ahead to performing analytics on all the instances, right now I have a postgres instance setup using RLS to separate data between tenants. Currently it's relatively trivial for me to see how users interact with the application just by writing a sql query and exporting the report.

Would the best tool to get a similar experience be using analytics-engine? Although I don't always know the data I'm after ahead of time (and pricing hasn't been announced for this yet).

Would it be feasible to create a report generating worker which loops through each of the D1 instances running a similar query and spits it out into a report. Or is there a better way?

EEpailes I'm thinking ahead to performing analytics on all the instances, right now I hav...

Matt Silverlock•10/2/23, 5:49 PM

What kind of data are you after? I would use Analytics Engine for "product events" (like what I think you are describing) and D1's own GraphQL API for "show me top-level usage by database: rows read, written, errors, etc"

kian•10/2/23, 6:47 PM

Usually within a week or two.

kian•10/2/23, 6:47 PM

The Cloudflare Workers runtime needs to make a new release using the latest workerd release/commit.

MMatt Silverlock What kind of data are you after? I would use Analytics Engine for "product event...

Epailes•10/2/23, 6:48 PM

Yeah this is more around product events, eg what is the most popular type of product category, what theme is the most used, how many products are hosted by the platform in total across all tenants?

MMatt Silverlock FYI: you will be able to create up to 50,000 D1 databases per account as of this...

Sameer Ali•10/2/23, 7:01 PM

THAT'S GREAT. is that going to be available for durable object only or we'll be able to use them in worker with dynamic binding?

Unsmart•10/2/23, 7:06 PM

Thats not dynamic bindings yet afaik you would have to pre bind each database or use the API to update the worker binding to add the new one. Probably best and easiest used with WfP

Unsmart•10/2/23, 7:07 PM

And not really sure how durable object come into play in your question

UUnsmart And not really sure how durable object come into play in your question

Max (@rozenmd)•10/3/23, 5:02 AM

like how DOs don't have to bind to individual objects

Unsmart•10/3/23, 5:04 AM

Yeah thats the nice part with DOs for sure but maybe im just reading into the wording too much but it seems like it was a bit of a different question

MMatt Silverlock FYI: you will be able to create up to 50,000 D1 databases per account as of this...

GuillaumeLakano•10/3/23, 9:34 AM

Do you known if this soft limit of 50,000 databases will be increased again later for the paid plans please?
I known it's possible to have kind of unlimited databases in WfP, but this require a business account of $200 / month if I understood correctly.
Our goal is to isolate 1 DB per user for security reason (like a sandbox)

GGuillaumeLakano Do you known if this soft limit of 50,000 databases will be increased again late...

itsmatteomanf•10/3/23, 9:36 AM

Matt will reply specifically for this case, but in general for Cloudflare soft limits can be increased for any account, but may require specific subscriptions to do so. I presume, but have no confirmation, that you can increase it just by asking Matt (while in beta, maybe even after) or opening a support ticket.

GGuillaumeLakano Do you known if this soft limit of 50,000 databases will be increased again late...

Epailes•10/3/23, 9:47 AM

Fyi ent contracts are closer to $2000/month last I checked

EEpailes Fyi ent contracts are closer to $2000/month last I checked

GuillaumeLakano•10/3/23, 9:48 AM

There is a Business plan at $200, but I don't known if WfP is only for Enterprises account or Business, but in all case even Business for us it's out of our budget

GuillaumeLakano•10/3/23, 9:50 AM

Generally, the security features of Cloudflare are free, as the goal is to have a better/safer internet. Our goal is only to add a extra security layer with isolate each DB per user, so in case of a successful attack, it's mitigated to only the user database

divby0•10/3/23, 10:00 AM

I upgraded my wrangler v2 to v3 and in that process I start getting Expression tree is too large (maximum depth 20)Expression tree is too large (maximum depth 20) for some queries. Those are not particularly complex, it's just like get x -> join this field -> join this field, so I'd guess this would be the depth 2 or 3?

It worked in v2 and it works on prod, so how comes it doesn't work when I locally upgrade to wrangler v3?

Here's the specific query:

  const teamArticles = await db.query.articles.findMany({
    where: ownerOrInTeam(articles.userId, authUser), // this is one equality check
    with: {
      pineconeReferences: true,
      outlineDrafts: { // if I do outlineDrafts: true, I am not over 20 anymore
        with: { // this seems to be the edge at which it goes over 20 seemingly
          draftPieces: true
        }
      }
    }
  });

  const teamArticles = await db.query.articles.findMany({
    where: ownerOrInTeam(articles.userId, authUser), // this is one equality check
    with: {
      pineconeReferences: true,
      outlineDrafts: { // if I do outlineDrafts: true, I am not over 20 anymore
        with: { // this seems to be the edge at which it goes over 20 seemingly
          draftPieces: true
        }
      }
    }
  });

This is drizzle syntax by the way if that helps.

divby0•10/3/23, 10:03 AM

Just to make it clear, this happens on several requests, not just this particular query.

const teamArticles = await db.query.articles.findMany({ where: ownerOrInTeam(articles.userId, authUser), // this is one equality check with: { pineconeReferences: true, outlineDrafts: { // if I do outlineDrafts: true, I am not over 20 anymore with: { // this seems to be the edge at which it goes over 20 seemingly draftPieces: true } } } });

Can I add data to a d1 database from workers and a python script?

Similar Threads

Similar Threads

Similar Threads