We fully transitioned from mongodb to d1 yesterday and it's plenty fast! The python script that runs

We fully transitioned from mongodb to d1 yesterday and it's plenty fast! The python script that runs on our server updates the DB while our pages functions act as the client which return the results on our api. Considering that many requests happen every minute which receive the same output how could we cache the response for example one minute in order to not burn row reads?

SSamantas5855 We fully transitioned from mongodb to d1 yesterday and it's plenty fast! The pyt...

Cyb3r-Jak3•10/20/23, 1:27 PM

Use the cache API to cache responses. The function will still run but will reduce usage of D1

Jonathan•10/20/23, 4:26 PM

Hey all, I'm new here. I'm noticing really slow performance using D1 (cold and hot runs) in other countries like Germany and Australia. I'm getting ~1200ms on cold starts and ~700ms on hot. Is this normal for the beta?

JJonathan Hey all, I'm new here. I'm noticing really slow performance using D1 (cold and h...

itsmatteomanf•10/20/23, 4:41 PM

Where is the D1 located? Or better yet, where have you created it from?

Original message was deleted

itsmatteomanf•10/20/23, 4:46 PM

True, I assumed the current beta database version, but yeah…

Jonathan•10/20/23, 4:49 PM

To clarify, my queries were just for reads.

Most recent version, I think? Just created the D1 on Oct 1st and testing with a deployment made yesterday. I think it was in us-east.. but not sure where to check the region.

JJonathan To clarify, my queries were just for reads. Most recent version, I think? Just ...

itsmatteomanf•10/20/23, 4:51 PM

It is the closest location, in which D1 is supported, from the location you hit when you create it. So if you are in us-east, it would have been created in and around there.

itsmatteomanf•10/20/23, 4:51 PM

So a 700ms latency, with the ocean crossing might make sense, even though it seems a bit high. How did you fetch it? Via a Worker?

itsmatteomanf•10/20/23, 4:52 PM

You should see actual database execution times in the query response json (under meta)

Iitsmatteomanf It is the closest location, in which D1 is supported, from the location you hit ...

Jonathan•10/20/23, 5:04 PM

It's via a worker. I'll have to check the meta. Should I expect their to be any regional or edge caching with D1 in the beta? In latency testing in US-Mich, Germany, NL, and Australia, I haven't seen faster latency than a singe region RDS micro instance.

JJonathan It's via a worker. I'll have to check the meta. Should I expect their to be any ...

itsmatteomanf•10/20/23, 5:06 PM

There will be read-replicas, but an ETA is currently unknown. You can do your own caching in the worker via the cache api

Iitsmatteomanf There will be read-replicas, but an ETA is currently unknown. You can do your ow...

Jonathan•10/20/23, 5:08 PM

Ah, got it that makes sense for now. I'd make a note in the d1 docs to mention this as a workaround, as it wasn't clear to me if I was going to roll out an app.

JJonathan Ah, got it that makes sense for now. I'd make a note in the d1 docs to mention t...

itsmatteomanf•10/20/23, 5:10 PM

Yeah, but it works and it won’t be needed in the long run (even though it’s reduce costs, so it makes sense all the time, tbh)

MMatt Silverlock There is not a transactions beta. What are you trying to do / how do you want to...

DaniFoldi•10/20/23, 8:39 PM

Hey Matt, I'll bump this message with another "use case", or more correctly, reason to implement - ORMs and libraries. Lots rely on transactions being a thing, and db.batch doesn't fully replace, as you can't split them and execute business logic inbetween queries.

Do you have sql transactions on the roadmap somewhere? MeowHeartCloudflare

parisholley•10/20/23, 11:20 PM

anyone ran into issues where migrations list works when using --local but not when trying to apply to prod? i just get empty output when running against prod (no errors). trying to migrate a new database for first time

Iitsmatteomanf There will be read-replicas, but an ETA is currently unknown. You can do your ow...

Paul•10/21/23, 5:57 AM

Oh, D1 isn’t actually on the edge just yet then? I had assumed the workers would be querying from the closest D1 instances, making for very fast reads. Isn’t that the main value prop of d1?

Paul•10/21/23, 6:01 AM

I’m not seeing anywhere on the website where it states that the read replicas are disabled. I was looking to launch Monday so this is terrible news for me. Anyone have info on this?

Old Man Maker•10/21/23, 8:50 AM

Would be great to see some work done on wrangler validation generally around naming conventions. One of my team created a table in the dashboard with a hyphen in it.

PPaul Oh, D1 isn’t actually on the edge just yet then? I had assumed the workers would...

itsmatteomanf•10/21/23, 9:02 AM

D1 is on the edge, it’s generated closest to where it’s generated. Read-replicas aren’t here yet,during the beta.

PPaul I’m not seeing anywhere on the website where it states that the read replicas ar...

itsmatteomanf•10/21/23, 9:03 AM

It will still work, it’s just not distributed. It’s a beta after all…

itsmatteomanf•10/21/23, 9:03 AM

It does seem weird it’s not written anywhere replicas don’t exist…

Iitsmatteomanf D1 is on the edge, it’s generated closest to where it’s generated. Read-replicas...

Jonathan•10/21/23, 11:22 AM

I think people expect D1 to behave more than Turso where DB read and writes are made only to a local db edge instance, which is asynchronously later sent to the primary region. This keeps writes and reads fast for the worker as it doesn't have to wait on the primary region (outside of cold starts sync).

Client applications can connect directly to a replica for read and write operations, but any writes are automatically forwarded to the primary.

Client applications can connect directly to a replica for read and write operations, but any writes are automatically forwarded to the primary.

https://docs.turso.tech/concepts

Concepts | Turso

Concepts for understanding how Turso works.

Iitsmatteomanf D1 is on the edge, it’s generated closest to where it’s generated. Read-replicas...

Paul•10/21/23, 11:36 AM

Ok, so you’re saying the main database, which is the single source of truth, exists in the location closest to where it was created?

JJonathan I think people expect D1 to behave more than Turso** where DB read and writes ar...

Paul•10/21/23, 11:37 AM

Yes this is what I expected! Will this be how D1 works once read replicas are live?

Iitsmatteomanf It will still work, it’s just not distributed. It’s a beta after all…

Paul•10/21/23, 11:38 AM

I get that it’s beta, but I wasn’t expecting the main value prop to be missing at the moment. I mean… I could have kept my Postgres database and just added caching, there’s no difference right?

Iitsmatteomanf It does seem weird it’s not written anywhere replicas don’t exist… 🤔

Paul•10/21/23, 11:39 AM

Yea, total gotcha. I’m bummed

JJonathan I think people expect D1 to behave more than Turso** where DB read and writes ar...

itsmatteomanf•10/21/23, 12:12 PM

As far as I know that's never been what the premise was. Single source of truth, edge, close to the first creation location + read-replicas spread around the world (not sure how this works, as I'm not on the team as you know).

PPaul Ok, so you’re saying the main database, which is the single source of truth, exi...

itsmatteomanf•10/21/23, 12:12 PM

Yes, and it always was planned and explained that way. Write will always go there.

PPaul I get that it’s beta, but I wasn’t expecting the main value prop to be missing a...

itsmatteomanf•10/21/23, 12:13 PM

You can still add caching, that is still true... and will always be reccomended, you pay per row read and written, so the less activity, the less it costs. And the Cache API on Workers is free.

itsmatteomanf•10/21/23, 12:15 PM

As far as read-replicas, I'm 100% sure it was said here multiple times, by many. I always just assumed it's in the docs somewhere, but I might have been wrong there.

itsmatteomanf•10/21/23, 12:15 PM

It's also a beta, which is clearly marked for non-production workloads. There are issues that happen sometimes.

itsmatteomanf•10/21/23, 12:22 PM

Gonna add that in no place, where D1 open betas where announced, did anybody say that read-replicas were a thing:

What’s next for D1?
We’re setting a clear goal: we want to make D1 “generally available” (GA) for production use-cases by early next year (Q1 2024). Although you can already use D1 without a waitlist or approval process, we understand that the GA label is an important one for many when it comes to a database (and as do we).

Between now and GA, we’re working on some really key parts of the D1 vision, with a continued focus on reliability and performance.

One of the biggest remaining pieces of that vision is global read replication, which we wrote about earlier this year. Importantly, replication will be free, won’t multiply your storage consumption, and will still enable session consistency (read-your-writes). Part of D1’s mission is about getting data closer to where users are, and we’re excited to land it.
— https://blog.cloudflare.com/d1-open-beta-is-here/

Paul•10/21/23, 12:24 PM

https://blog.cloudflare.com/introducing-d1/

The Cloudflare Blog

Announcing D1: our first SQL database

Today, we’re excited to announce D1, Cloudflare’s first SQL database, designed for Cloudflare Workers

Paul•10/21/23, 12:24 PM

I think this might be the one that got me

itsmatteomanf•10/21/23, 12:25 PM

Yeah, in the announcements they talk about read-replicas, but in both the open alpha, and open beta it's mentioned replication isn't here yet.

Paul•10/21/23, 12:25 PM

That cross referenced with the “Limits” page

itsmatteomanf•10/21/23, 12:26 PM

The open beta blog post mentions a GA timeframe of Q1 2024, in which that is expected to exist.

Paul•10/21/23, 12:26 PM

So really the only way to find out is dig through discord, or just so happen to see it like I did

Paul•10/21/23, 12:26 PM

lol

PPaul So really the only way to find out is dig through discord, or just so happen to ...

itsmatteomanf•10/21/23, 12:26 PM

Well, no. Also in all open alpha and open beta blog posts it's mentioned it's not yet included...

Paul•10/21/23, 12:27 PM

Oh really… I didn’t notice it was mentioned as not included

itsmatteomanf•10/21/23, 12:27 PM

Yeah, it's in there.

itsmatteomanf•10/21/23, 12:29 PM

In the Open Alpha announcement: "The biggest pieces that have been disabled for this alpha release are replication and JavaScript transaction support."
In the Open Beta announcement: "One of the biggest remaining pieces of that vision is global read replication, which we wrote about earlier this year."

Paul•10/21/23, 12:29 PM

Yep, I see that now. Thanks

Paul•10/21/23, 12:30 PM

Luckily I abstracted away the database behind a repo layer… so I can swap it out

Iitsmatteomanf Yeah, it's in there.

Paul•10/21/23, 12:31 PM

Do you know if reads from a read replica will be charged as well? Or is that kind of “caching out of the box”?

Paul•10/21/23, 12:32 PM

That would be awesome

PPaul Do you know if reads from a read replica will be charged as well? Or is that kin...

itsmatteomanf•10/21/23, 12:32 PM

As far as I expect, as I haven't had more details than you do, they will be charged normally.

itsmatteomanf•10/21/23, 12:32 PM

You won't be charged more for the read replicas themselves. So no storage nor replications costs.

Paul•10/21/23, 12:33 PM

So that’s what they mean… well here’s to hoping reads from it are free as well

That would be insane for adoption

itsmatteomanf•10/21/23, 12:34 PM

I really doubt that... but hey, the Cache API still exists

We fully transitioned from mongodb to d1 yesterday and it's plenty fast! The python script that runs

Similar Threads

Similar Threads

Similar Threads