Cloudflare Developers•2y ago

As long as it's allowed to proxy through workers that's fine with me

EliaOP•2/13/24, 12:52 AM

Any chance they'll upgrade the limits before GA? (e.g. talking max storage, SQL operators, etc.)

EElia As long as it's allowed to proxy through workers that's fine with me

Vy•2/13/24, 1:15 AM

We're working on max storage. What additional SQL operators ar eyou looking for?

MMax (@rozenmd)fts5 is available

down•2/13/24, 7:11 AM

I know, but it's nowhere near as sophisticated as a search engine.

ZZiga Zajc What is the average latency added when we use D1 thru HTTP API?

Hello, I’m Allie!•2/13/24, 7:19 AM

Depends on where you are

Hello, I’m Allie!•2/13/24, 7:19 AM

All requests go through Core(iirc), so if you are far away then it takes longer

Hello, I’m Allie!•2/13/24, 7:20 AM

Even worse if your DB is close to you

ZZiga Zajc for example Hetzner datacenter in Nuremberg and Falkenstein (Germany)

Hello, I’m Allie!•2/13/24, 7:24 AM

Run it through a Worker, would be a lot faster

ZZiga Zajc The problem is that I'm making this project open source, so it can be self-hoste...

Hello, I’m Allie!•2/13/24, 7:28 AM

I would just note that D1 really isn’t meant to be accessed that way, and thus I doubt it will every be very performant

ZZiga Zajc wait why D1 HTTP API wasn't build on Cloudflare Workers from the start?

Hello, I’m Allie!•2/13/24, 7:29 AM

It is, afaik. But the entirety of api.cloudflare.com is routed to Core

ZZiga Zajc so users / self-hosters wouldn't need to deploy their own D1 proxy on workers

Hello, I’m Allie!•2/13/24, 7:30 AM

Because D1 is a DevPlat DB, which means it performs best when used from other DevPlat products

Hello, I’m Allie!•2/13/24, 7:30 AM

Like Workers/Functions

ZZiga Zajc I could probably completely replicate D1 HTTP API in workers. So normal self-hos...

Hello, I’m Allie!•2/13/24, 7:36 AM

Wait, do you mean you would make requests for them?

HHello, I’m Allie!It is, afaik. But the entirety of api.cloudflare.com is routed to Core

Max (@rozenmd)•2/13/24, 7:44 AM

It isn't (yet) - it was easier to take an existing approach we used often than to start something new, since the aim is to make the Workers platform better, not to make a standalone db

Max (@rozenmd)•2/13/24, 7:44 AM

D1's API will be run from Workers :soontm:

MMax (@rozenmd)It isn't (yet) - it was easier to take an existing approach we used often than t...

Hello, I’m Allie!•2/13/24, 7:46 AM

TIL

Hello, I’m Allie!•2/13/24, 7:46 AM

Might also help to be able to hit EuroCore, even when PDX isn't down

HHello, I’m Allie!Might also help to be able to hit EuroCore, even when PDX isn't down

Max (@rozenmd)•2/13/24, 7:47 AM

It runs in both in an active/active config

Hello, I’m Allie!•2/13/24, 7:48 AM

Nvm

MMax (@rozenmd)D1's API will be run from Workers :soontm:

Mozzy•2/13/24, 9:27 AM

Does this mean the HTTP API will be comparable in perfprmance to creating a proxy worker ourselves?

MMozzy Does this mean the HTTP API will be comparable in perfprmance to creating a prox...

Max (@rozenmd)•2/13/24, 9:30 AM

The API has unavoidable overhead that you can avoid by creating your own Worker

Max (@rozenmd)•2/13/24, 9:30 AM

but it'll be a lot more comparable than now

ZZiga Zajc awesome, what about maximum database size limit. Will this be completely removed...

Max (@rozenmd)•2/13/24, 9:51 AM

Hello, I’m Allie!•2/13/24, 9:52 AM

https://canary.discord.com/channels/595317990191398933/1199016695491219469/1203006777621356544

Max (@rozenmd)•2/13/24, 9:53 AM

D1 is built on Durable Objects - it's easy for us to support millions of databases and load them across the world quickly. Big databases, less so

ZZiga Zajc this comes with D1 GA release?

Max (@rozenmd)•2/13/24, 10:52 AM

don't know yet

Tony Leung•2/13/24, 2:04 PM

hi, will D1 have error code like mysql or postgresql (like in this document: https://www.postgresql.org/docs/current/errcodes-appendix.html) ? Currently I'm handling errors by parsing the error messages, which is kinda intuitive.

Ɽ

ⱤØ₥₳₦•2/13/24, 2:45 PM

Is it possible to build analytics system in D1

Like i have 2k + posts and each has unique id.

So i want to use D1 as database for pageviews system (hourly)

So i can see trends hourly, monthly and yearly

ⱤⱤØ₥₳₦Is it possible to build analytics system in D1 Like i have 2k + posts and each ...

Max (@rozenmd)•2/13/24, 3:29 PM

if you have under 400 writes per second, yep, but strongly recommend Workers Analytics Engine for that use case instead

MMax (@rozenmd)if you have under 400 writes per second, yep, but strongly recommend Workers Ana...

Ɽ

ⱤØ₥₳₦•2/13/24, 3:33 PM

More than 3k cause on average 3k users are live on my site. I thought i should use google analytics with API

but that's too complex

ⱤⱤØ₥₳₦More than 3k cause on average 3k users are live on my site. I thought i should u...

Max (@rozenmd)•2/13/24, 3:41 PM

check this out: https://developers.cloudflare.com/analytics/analytics-engine/
we do hundreds of millions of requests per second through it, and you can query it with SQL

Chaika•2/13/24, 3:42 PM

could always do something silly where you have a worker with a cron trigger every hour or so to pull aggregated stats down into D1

Chaika•2/13/24, 3:43 PM

then you'd have 90 days or nice adaptive sampled analytics for some graphs and longer term aggregated

MMax (@rozenmd)check this out: https://developers.cloudflare.com/analytics/analytics-engine/ we...

Ɽ

ⱤØ₥₳₦•2/13/24, 4:04 PM

That's insane. I'll try this

thanks for reply

CChaika then you'd have 90 days or nice adaptive sampled analytics for some graphs and l...

Ɽ

ⱤØ₥₳₦•2/13/24, 4:04 PM

Thanks ma'am

Luka•2/13/24, 6:48 PM

Hey @Max (@rozenmd) , I underestimate the rate of growth of my database. I cannot export using Nora's tool because I hit all types of limits and I will hit the upper filesize limit in the next month (I estimate, if not sooner).

I feel trapped and the only reasonable way is to write an exporter and export each row to remote using API. Would Cloudflare be able to export a .sqlite dump or temporarily lift any limits on my account to complete it?

As long as it's allowed to proxy through workers that's fine with me

Similar Threads

Similar Threads

Similar Threads