Cloudflare Developers•15mo ago

Speaking of... do we know roughly when read replication may see a release (in any form, eg. private

Speaking of... do we know roughly when read replication may see a release (in any form, eg. private beta)?

JJacob Marshall Speaking of... do we know roughly when read replication may see a release (in an...

Chaika•10/14/24, 12:42 AM

hmm I think last date I heard was Q3 d1-database

Chaika•10/14/24, 12:42 AM

well, "Targeting"

Chaika•10/14/24, 12:42 AM

then "private beta soon" last month d1-database

Jacob MarshallOP•10/14/24, 12:44 AM

Oh thanks, @Chaika. For a second there when you said Q3 I thought you meant next year haha (considering we're already in Q4)

Jacob MarshallOP•10/14/24, 12:44 AM

I was prepared to cry ;P

bennycondon•10/14/24, 1:11 AM

Does D1 support escaping single quotes by doubling up the single quotes? e.g. INSERT INTO test VALUES ('This''ll work just fine.')INSERT INTO test VALUES ('This''ll work just fine.').

It seems to work fine when I run npx wrangler d1 execute <database> --file=<file>npx wrangler d1 execute <database> --file=<file> for local development.

But gives me an internal error when I run npx wrangler d1 execute <database> --file=<file> --remotenpx wrangler d1 execute <database> --file=<file> --remote for production.

Georg Obermayr•10/14/24, 4:03 PM

I'm having some performances troubles with D1 databases. I have a Worker that is bound to a D1 database and using smart placement. We are on a enterprise plan. I have a very simple API in that Worker that just selects some items by an id in the route, parses some JSON fields and returns the data. With that I have response times starting at 500 ms, sometimes up to 2s! We also did some load testing with 500 users, and with that the response time exploded to 20 seconds and more. Obvously we can't use that in production. I'm requesting from Munic/Germany, and the response says remote-HAM for CF-placement. Is there anything that can be done?

o7•10/14/24, 9:11 PM

Will the D1 Dynamic Binding future integration include a batch API to delete more than 1000 (worker subrequest limit) databases in a single go? This would be a handy feature for deleting many expired user accounts efficiently in a cron worker, as my usecase.

AsyncBanana•10/15/24, 12:15 AM

I am getting the error {"D1_RESET_DO":true}{"D1_RESET_DO":true} whenever I try to execute a SQL file on a remote D1 database. No errors appear when I execute the same file on a local database, and the source file is from a working D1 local database. Any ideas what could be happening?

ilsubyeega•10/15/24, 11:06 AM

Is there any way to optimize the cost of writes when indexing? It seems like each index adds one write. Not that it is really important, but I'm curious.

Iilsubyeega Is there any way to optimize the cost of writes when indexing? It seems like eac...

lambrospetrou•10/15/24, 11:23 AM

That's expected and documented (your writes need to update your indices). Optimizing that out means removing your index, but then you make your reads much slower/costlier, so it depends on the frequency between reads and writes and what's more cost-effective for your case.
I wouldn't recommend removing your indices though since latencies could go high depending on what your queries are, so it's not just a cost optimization.

ilsubyeega•10/15/24, 11:24 AM

thanks :D

Hhannes More data needed, too fuzzy: parsing JSON fields how? do you read whole JSON obj...

Georg Obermayr•10/15/24, 4:15 PM

I mostly did JSON.parse, but in this test scenario it was a single small field in a single record. I also tried to just do the query (Select * from table where id=?Select * from table where id=?) and not parse/return anything, just testing the timing. Still a very high latency and response time.

nato•10/15/24, 8:10 PM

is d1 faster when used with workers compared to cloudflare pages http client? considering if I should use opennext or next-on-pages

Nnato is d1 faster when used with workers compared to cloudflare pages http client? co...

Sameer Ali•10/15/24, 8:49 PM

yes

nato•10/15/24, 8:55 PM

thanks!

Alexander Hupfer•10/15/24, 9:40 PM

I'm trying to use prisma with workers, but running into bundle size issues. Is there something I can do?

Dean•10/16/24, 2:37 AM

One thing I’d love to see in d1 is some sort of basic metadata option. Ie for each db be able to put a note or something about the service (what it’s used for) or just some simple json.

DangerZone•10/16/24, 3:34 AM

Is it possible to increase the maximum database size from 10gb to about 100gb for an enterprise plan?

DDangerZone Is it possible to increase the maximum database size from 10gb to about 100gb fo...

Cyb3r-Jak3•10/16/24, 3:35 AM

Pretty sure it is not.

DangerZone•10/16/24, 3:38 AM

There's a form on the limitation page in the docs, just asking if it is possible since the form is there

DDangerZone There's a form on the limitation page in the docs, just asking if it is possible...

Chaika•10/16/24, 3:41 AM

They've said before they currently don't increase that: d1-database and if they did I would expect maybe 20gb, not 100gb as called out in his msg

Noitidart•10/16/24, 11:26 AM

Is it possible to query across multiple d1 databases? Like thousands at once?

Was considering separate db per customer, closest to region, but for some things I need to run on all. Also migrations needs to run on all.

NNoitidart Is it possible to query across multiple d1 databases? Like thousands at once? W...

lambrospetrou•10/16/24, 12:26 PM

Not yet out of the box, you have to write the cross-db query logic.

Also migrations needs to run on all.

How would you like this to work, say you could have exactly the system you need, what would it look like? What would it do for you, and what would you like to do?

Llambrospetrou Not yet out of the box, you have to write the cross-db query logic. > Also mig...

Noitidart•10/16/24, 12:29 PM

Thanks very much for asking. I'm still trying to solve the problem of making writes go to nearest db to customer. So if My customer in Australia, it should write to db there. Even if the worker is running in USA (most will be from within customer region, 0.1% writes will be customer support so from USA)

Ideally I still want 1 db. As multi is really hard to work with. As multiple cases of needing to have data on single db, like shared users across customers.

Noitidart•10/16/24, 12:36 PM

I think my ideal system in my read and write I specify region, and i grantee my IDs are unique (uuid not incrementing nimber) so it's strongly consistent provided I told it same region. And it's strongly consistent everywhere else within X seconds for propagation.

Larry•10/16/24, 12:36 PM

@DangerZone , I suggest you look at the new beta SQLite API inside of DOs. The limit there is currently 1GB but it will go up to 10GB at the end of beta. So, it won't help you scale up a single DB size, but it might be more convenenient than "thousands" of D1 DBs because DO code runs in the same thread as the DB. Dynamic binding would be necessary to manage that many. However, w/ DOs you wouldn't need dynamic binding. Also, if you want to aggregate a query result accross 1000s of DBs, you can do any needed manipulations and reduction locally, then send those reduced results to a central aggregator.

NNoitidart I think my ideal system in my read and write I specify region, and i grantee my ...

Noitidart•10/16/24, 12:42 PM

Further thinking on my ideal system. If no region specified, then it works as is today, it writes to default region, and waits till its strongly replicated acrosss all regions, so its strongly consistent regardless of region.

Not sure if this is possible or planned, but this would let me global app work fast globally. As my bottleneck right now is my postgres db for writes as a lot of admin work happens in various parts of world. (reads are fine its replicated cached)

LLarry @DangerZone , I suggest you look at the new beta SQLite API inside of DOs. The l...

DangerZone•10/16/24, 1:13 PM

That might work, SQLite does support some powerful fts capabilities.
Our use-case is to create a full-text-search service on addresses tho, so we will have to be creative with our sharding mechanism (I'm still unsure about the sharding algo to support that tho)

Eeva2000 So what options are there when someone reaches their D1 database size limits? Ju...

Chaika•10/16/24, 1:20 PM

The d1 team currently wants you to shard your data among many D1 DBs rather then one big one. Could be sharded by whatever makes sense for your app. By users, by company, etc
d1-database

Hhannes ok, weird. You are sure that this is not an old alpha D1 DB? And that it's locat...

Georg Obermayr•10/16/24, 1:24 PM

Well, I recreated the DB a couple of times in the previous weeks so I think it is the current version. Honestly I don't know to determine the location of the DB, so this is from the location of the bound worker with smart placement. The first DB I created was in Madrid, the second in Paris, and now Hamburg. I tried with location hints and without.
In the previous days I did a migration to Neon (https://neon.tech) because we had a launch date and this was very urgent. I used the same worker, just refactored the database part from D1 to Neons HTTP-Proxy and Websocket APIs. This is now blazing fast!
Yeah, I will definitly talk with Cloudflare and raise our issues. Thank you!

Neon

Neon Serverless Postgres — Ship faster

The database you love, on a serverless platform designed to help you build reliable and scalable applications faster.

kw•10/16/24, 9:38 PM

Potential Bug report:
db id: 75ef58ba-b54f-411d-bec1-a184986f143c,
{"TimestampMs":1729112916377},{"Level":"error","Message":["Error: D1_ERROR: Error: Internal error while starting up D1 storage caused object to be reset."],"TimestampMs":1729112932597}

as of now, db recovered automatically.

1984 Ford Laser•10/17/24, 12:37 AM

Has anyone tried using this to get dynamic D1 bindings? How far is this from being supported officially?
https://github.com/nora-soderlund/cloudflare-dynamic-bindings

GitHub

GitHub - nora-soderlund/cloudflare-dynamic-bindings: A package that...

A package that aims to provide a single solution for managing dynamic binds for Cloudflare Workers. - nora-soderlund/cloudflare-dynamic-bindings

IEatBeans•10/17/24, 3:38 AM

I am exporting the database ina github action, and its been working but the last few days it started failing and just saying "Error". No other info.

Does anyone know how I can find more info? Since its a gh action idk how to view that log file.

Walshy•10/17/24, 4:29 PM

what's the sql query?

Walshy•10/17/24, 4:38 PM

yes, share the schema.sql

Speaking of... do we know roughly when read replication may see a release (in any form, eg. private

Similar Threads

Similar Threads

Similar Threads