Thanks for the response, I assume this is under a bindings API?

Old Man MakerOP•10/19/23, 7:50 AM

Actually I see worker binding api is depreciated

Old Man MakerOP•10/19/23, 8:04 AM

I think you're referring to the patch functions for worker script API? That's about all I could find. The problem with that is we are limited to the number of bindings as far as I know to about 5000 per worker. So this would not be a suitable solution if we have say 20k databases.
If the only option is to query a DB with REST calls, I assume there is a latency penalty as opposed to a binding call if executed from the same worker?

Michał•10/19/23, 8:16 AM

Yes, that's what I was referring to. Why do you have 5000 databases binded to a single worker though? It sounds curious, what's the usecase?

MMichał Yes, that's what I was referring to. Why do you have 5000 databases binded to a ...

Old Man MakerOP•10/19/23, 8:29 AM

It fits the overall use case of D1 which is intended to open up database-per-user use cases. In our use case, when a customer creates a new project, we generate several databases for them. We need to to this, because of the 2Gb limit / database. If we have several thousand projects created, single DBs sizes are exceeded quickly, as well our customers want their data isolated. With even just 1000 customers, we could end up with around ~35K databases.

Managing and querying these with REST is fine for the most part, except for a very specific scenario involving AI where the latency of a REST call would be an issue - but that worker making the query, needs to be able to access any project DB, so a scenario where we could call by name or ID would be great.

Michał•10/19/23, 8:30 AM

We deploy a copy of the worker along with a database for each client

Not sure if that helps your case though

MMichał We deploy a copy of the worker along with a database for each client 🙂 Not sure...

Old Man MakerOP•10/19/23, 8:30 AM

hmmm - thats very interesting and I had not considered that - thank you so much for that insight

Old Man MakerOP•10/19/23, 8:31 AM

You just use an API to duplicate it?

Old Man MakerOP•10/19/23, 8:32 AM

I was just noticing there is a limit of 500 workers per account

Michał•10/19/23, 8:32 AM

It's more complex than that in our case but more or less yeah

MMichał It's more complex than that in our case but more or less yeah

Old Man MakerOP•10/19/23, 8:33 AM

Thanks for that info - really appreciate it

OOld Man Maker I was just noticing there is a limit of 500 workers per account

Michał•10/19/23, 8:37 AM

I think that can be increased on individual basis but I can't speak for Cloudflare, not affiliated or anything

GuillaumeLakano•10/19/23, 1:58 PM

Hi ! The LIMIT in a UPDATE/DELETE request doesn't seems to be supported, it's depend of the configuration of SQLite ( https://www.sqlite.org/compile.html#enable_update_delete_limit ), is it possible to enable it please?

OOld Man Maker Thanks for the response, I assume this is under a bindings API?

Matt Silverlock•10/19/23, 2:47 PM

PATCH script settings

OOld Man Maker I think you're referring to the patch functions for worker script API? That's ab...

Matt Silverlock•10/19/23, 2:47 PM

https://developers.cloudflare.com/api/operations/worker-script-patch-settings

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

OOld Man Maker Thanks for the response, I assume this is under a bindings API?

Matt Silverlock•10/19/23, 2:48 PM

e.g.

➜  curl -v --request PATCH \
  --url "https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/workers/scripts/postgres-js-hyperdrive/settings" \
  --header "Authorization: Bearer ${CF_API_TOKEN}" \
  --header 'Content-Type: multipart/form-data' --form 'settings={"bindings": [{
        "id": "3dfe2fcb-b98b-48f8-88d5-704b74af7472","name":"test-db-api",
        "binding": "D1_DB",
        "type": "d1"
      }]}'

➜  curl -v --request PATCH \
  --url "https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/workers/scripts/postgres-js-hyperdrive/settings" \
  --header "Authorization: Bearer ${CF_API_TOKEN}" \
  --header 'Content-Type: multipart/form-data' --form 'settings={"bindings": [{
        "id": "3dfe2fcb-b98b-48f8-88d5-704b74af7472","name":"test-db-api",
        "binding": "D1_DB",
        "type": "d1"
      }]}'

Matt Silverlock•10/19/23, 2:48 PM

This is an interim solution, for what it's worth. As we've called out before, we're working on a way to bind to a group of resources that can grow/shrink over time. This won't be "next month" though, but before D1 GA.

MMatt Silverlock This is an interim solution, for what it's worth. As we've called out before, we...

Old Man MakerOP•10/19/23, 2:58 PM

Thanks for that information. I'll do some tests around this. As @Michał z typeofweb.com mentioned his approach, my very real fear is discovering an issue in the root worker, and then needing to update N workers who were copied.

Samantas5855•10/19/23, 3:01 PM

While implementing D1 on Pages Functions I copied the example from here to get the first element of the table and it seems to take about 1.5s

Bindings · Cloudflare Pages docs

A binding enables your Pages Functions to interact with resources on the Cloudflare developer platform. Use bindings to integrate your Pages Functions …

Samantas5855•10/19/23, 3:02 PM

Why is it so slow? Is it possible to speed it up?

schwartzmj•10/19/23, 3:04 PM

I see Backups are legacy and "Time Travel" is the new thing. Am I not able to download a local backup of my database now? You could with the legacy method but not seeing a way with Time Travel

zegevlier•10/19/23, 3:05 PM

It is currently not possible to download D1 databases that are using the new backend (=that have time travel). There is a community project that lets you do that though

zegevlier•10/19/23, 3:06 PM

I haven't used it myself, but here's a link: http://github.com/nora-soderlund/cloudflare-d1-backups

GitHub

GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a ...

A package to dump a copy of your experimental D1 database through a worker. - GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a copy of your experimental D1 database through a worker.

Zzegevlier It is currently not possible to download D1 databases that are using the new bac...

schwartzmj•10/19/23, 3:07 PM

alright thanks... any idea if its planned to be able to download your database at some time in the future? like, confirmed

zegevlier•10/19/23, 3:26 PM

I have not seen any confirmation about it, and a quick search also did not show any

OOld Man Maker Thanks for that information. I'll do some tests around this. As @Michał z typeof...

Matt Silverlock•10/19/23, 3:30 PM

Right now you can bind up to ~5000 databases per Worker (conservatively), so I suspect this isn't an immediate problem just yet / before we have a better solution.

MMatt Silverlock Right now you can bind up to ~5000 databases per Worker (conservatively), so I s...

Old Man MakerOP•10/19/23, 3:34 PM

Thanks Matt. Given that the binding is added, can I access it with a key?
E.g const result = env[binding_name].prepare(,,,,)?

Old Man MakerOP•10/19/23, 3:35 PM

The binding name will be a parameter passed into the worker

Old Man MakerOP•10/19/23, 4:00 PM

I'm actually thinking I might be going about this all wrong for this particular use case - in D1 we've got a hammer, so everything looks like a nail.
In this instance, I think I may be better off using KV as the storage mechanism
This has been a really interesting exercise in considerations and thanks to everyone who has put thoughts, ideas and solutions into it. Much appreciated

OOld Man Maker Thanks Matt. Given that the binding is added, can I access it with a key? E.g c...

Matt Silverlock•10/19/23, 4:07 PM

We're still designing the interface, but effectively being able to address databases/resources by name/ID at runtime.