Cloudflare Developers•3y ago

I was just noticing there is a limit of 500 workers per account

Michał•10/19/23, 8:32 AM

It's more complex than that in our case but more or less yeah

MMichał It's more complex than that in our case but more or less yeah

Old Man MakerOP•10/19/23, 8:33 AM

Thanks for that info - really appreciate it

OOld Man Maker I was just noticing there is a limit of 500 workers per account

Michał•10/19/23, 8:37 AM

I think that can be increased on individual basis but I can't speak for Cloudflare, not affiliated or anything

GuillaumeLakano•10/19/23, 1:58 PM

Hi ! The LIMIT in a UPDATE/DELETE request doesn't seems to be supported, it's depend of the configuration of SQLite ( https://www.sqlite.org/compile.html#enable_update_delete_limit ), is it possible to enable it please?

OOld Man Maker Thanks for the response, I assume this is under a bindings API?

Matt Silverlock•10/19/23, 2:47 PM

PATCH script settings

OOld Man Maker I think you're referring to the patch functions for worker script API? That's ab...

Matt Silverlock•10/19/23, 2:47 PM

https://developers.cloudflare.com/api/operations/worker-script-patch-settings

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

OOld Man Maker Thanks for the response, I assume this is under a bindings API?

Matt Silverlock•10/19/23, 2:48 PM

e.g.

➜  curl -v --request PATCH \
  --url "https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/workers/scripts/postgres-js-hyperdrive/settings" \
  --header "Authorization: Bearer ${CF_API_TOKEN}" \
  --header 'Content-Type: multipart/form-data' --form 'settings={"bindings": [{
        "id": "3dfe2fcb-b98b-48f8-88d5-704b74af7472","name":"test-db-api",
        "binding": "D1_DB",
        "type": "d1"
      }]}'

➜  curl -v --request PATCH \
  --url "https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/workers/scripts/postgres-js-hyperdrive/settings" \
  --header "Authorization: Bearer ${CF_API_TOKEN}" \
  --header 'Content-Type: multipart/form-data' --form 'settings={"bindings": [{
        "id": "3dfe2fcb-b98b-48f8-88d5-704b74af7472","name":"test-db-api",
        "binding": "D1_DB",
        "type": "d1"
      }]}'

Matt Silverlock•10/19/23, 2:48 PM

This is an interim solution, for what it's worth. As we've called out before, we're working on a way to bind to a group of resources that can grow/shrink over time. This won't be "next month" though, but before D1 GA.

MMatt Silverlock This is an interim solution, for what it's worth. As we've called out before, we...

Old Man MakerOP•10/19/23, 2:58 PM

Thanks for that information. I'll do some tests around this. As @Michał z typeofweb.com mentioned his approach, my very real fear is discovering an issue in the root worker, and then needing to update N workers who were copied.

Samantas5855•10/19/23, 3:01 PM

While implementing D1 on Pages Functions I copied the example from here to get the first element of the table and it seems to take about 1.5s

Bindings · Cloudflare Pages docs

A binding enables your Pages Functions to interact with resources on the Cloudflare developer platform. Use bindings to integrate your Pages Functions …

Samantas5855•10/19/23, 3:02 PM

Why is it so slow? Is it possible to speed it up?

schwartzmj•10/19/23, 3:04 PM

I see Backups are legacy and "Time Travel" is the new thing. Am I not able to download a local backup of my database now? You could with the legacy method but not seeing a way with Time Travel

zegevlier•10/19/23, 3:05 PM

It is currently not possible to download D1 databases that are using the new backend (=that have time travel). There is a community project that lets you do that though

zegevlier•10/19/23, 3:06 PM

I haven't used it myself, but here's a link: http://github.com/nora-soderlund/cloudflare-d1-backups

GitHub

GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a ...

A package to dump a copy of your experimental D1 database through a worker. - GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a copy of your experimental D1 database through a worker.

Zzegevlier It is currently not possible to download D1 databases that are using the new bac...

schwartzmj•10/19/23, 3:07 PM

alright thanks... any idea if its planned to be able to download your database at some time in the future? like, confirmed

zegevlier•10/19/23, 3:26 PM

I have not seen any confirmation about it, and a quick search also did not show any

OOld Man Maker Thanks for that information. I'll do some tests around this. As @Michał z typeof...

Matt Silverlock•10/19/23, 3:30 PM

Right now you can bind up to ~5000 databases per Worker (conservatively), so I suspect this isn't an immediate problem just yet / before we have a better solution.

MMatt Silverlock Right now you can bind up to ~5000 databases per Worker (conservatively), so I s...

Old Man MakerOP•10/19/23, 3:34 PM

Thanks Matt. Given that the binding is added, can I access it with a key?
E.g const result = env[binding_name].prepare(,,,,)?

Old Man MakerOP•10/19/23, 3:35 PM

The binding name will be a parameter passed into the worker

Old Man MakerOP•10/19/23, 4:00 PM

I'm actually thinking I might be going about this all wrong for this particular use case - in D1 we've got a hammer, so everything looks like a nail.
In this instance, I think I may be better off using KV as the storage mechanism
This has been a really interesting exercise in considerations and thanks to everyone who has put thoughts, ideas and solutions into it. Much appreciated

OOld Man Maker Thanks Matt. Given that the binding is added, can I access it with a key? E.g c...

Matt Silverlock•10/19/23, 4:07 PM

We're still designing the interface, but effectively being able to address databases/resources by name/ID at runtime.

itsmatteomanf•10/19/23, 4:38 PM

@Matt my D1 db seems to have vanished...

546787dd-35d5-4822-9b50-36fa0a4310c2

546787dd-35d5-4822-9b50-36fa0a4310c2

Iitsmatteomanf @Matt my D1 db seems to have vanished... `546787dd-35d5-4822-9b50-36fa0a4310c2` ...

Matt Silverlock•10/19/23, 4:44 PM

Account ID please

Iitsmatteomanf @Matt my D1 db seems to have vanished... `546787dd-35d5-4822-9b50-36fa0a4310c2` ...

Matt Silverlock•10/19/23, 4:45 PM

And does vanished mean: "it doesn't respond to queries", "it doesn't show up in the API", or something else?

MMatt Silverlock Account ID please

itsmatteomanf•10/19/23, 4:45 PM

275fcefacfae5b20fe129cbac8269aa4275fcefacfae5b20fe129cbac8269aa4

MMatt Silverlock And does vanished mean: "it doesn't respond to queries", "it doesn't show up in ...

itsmatteomanf•10/19/23, 4:46 PM

Failed to reach database. Please try again later.

itsmatteomanf•10/19/23, 4:46 PM

Both UI and Workers (which returns D1_ERROR: internal errorD1_ERROR: internal error)

Iitsmatteomanf Both UI and Workers (which returns `D1_ERROR: internal error`)

Matt Silverlock•10/19/23, 4:47 PM

OK, so not "disappeared" - somehow stuck (still bad, but very different concern). Let us take a look.

MMatt Silverlock OK, so not "disappeared" - somehow stuck (still bad, but very different concern)...

itsmatteomanf•10/19/23, 4:48 PM

Thank you... it was growing in size, but shouldn't have hit the 2GB limit, at all. It seem to have started failing yesterday, like ~30 hrs ago.

Iitsmatteomanf Thank you... it was growing in size, but shouldn't have hit the 2GB limit, at al...

Matt Silverlock•10/19/23, 4:49 PM

Last I can see is 789MB

MMatt Silverlock Last I can see is 789MB

itsmatteomanf•10/19/23, 4:49 PM

Yeah, aligns with what I expected, yes.

Matt Silverlock•10/19/23, 4:49 PM

OK. Looking into why.

itsmatteomanf•10/19/23, 4:49 PM

Mostly in one single table.

itsmatteomanf•10/19/23, 4:49 PM

Thank you

niconiahi•10/19/23, 6:43 PM

any recommendation on how to visualize my d1 database locally? in preference something completely open source and simplistic. Thanks!

Iitsmatteomanf Mostly in one single table.

Matt Silverlock•10/19/23, 8:26 PM

Should be fixed now.

MMatt Silverlock Should be fixed now.

itsmatteomanf•10/19/23, 8:31 PM

Looks like it is! Thank you MeowHeartCloudflare

itsmatteomanf•10/19/23, 8:32 PM

Any insights on what happened, that you are willing to share?

Iitsmatteomanf Any insights on what happened, that you are willing to share? 🙂

Matt Silverlock•10/19/23, 8:33 PM

Not yet. We restored and are still working on the root cause.

SnowyYANG•10/19/23, 10:08 PM

d1 supports how many data type? when I create the table there is even no varchar, only text.

SSnowyYANG d1 supports how many data type? when I create the table there is even no varchar...

Cyb3r-Jak3•10/19/23, 10:11 PM

It should be all sqlite data types which are NULL, INTEGER, READ, TEXT and BLOB

SnowyYANG•10/19/23, 10:12 PM

so d1 is sqlite?

Cyb3r-Jak3•10/19/23, 10:12 PM

Yes

SnowyYANG•10/19/23, 10:12 PM

thanks

SnowyYANG•10/19/23, 10:17 PM

I have a large complex MySQL application (like erp), is it smart to migrate to D1?

Cyb3r-Jak3•10/19/23, 10:27 PM

How are you connecting to the MySQL DB? There isn't a connection string that you can use to connect to a D1 DB externally, done through a worker or HTTP

SnowyYANG•10/19/23, 11:07 PM

I originally plan to use PHP+MySQL. If I migrate MySQL to D1 then I have to migrate PHP to Worker.

SnowyYANG•10/20/23, 4:13 AM

tested. D1 performance is not good.

SSnowyYANG tested. D1 performance is not good.

Hello, I’m Allie!•10/20/23, 6:33 AM

Are you using the API? That is usually much slower

HHello, I’m Allie!Are you using the API? That is usually much slower

SnowyYANG•10/20/23, 6:51 AM

yes, I am using Page Functions to insert data into D1

I was just noticing there is a limit of 500 workers per account

Similar Threads

Similar Threads

Similar Threads