Cloudflare Developers•2y ago

https://developers.cloudflare.com/workers/platform/limits/ free gets 1 mb max size, paid gets 10 mb

https://developers.cloudflare.com/workers/platform/limits/ free gets 1 mb max size, paid gets 10 mb max workers size

zensin•7/26/24, 1:19 AM

ooh

ChaikaOP•7/26/24, 1:19 AM

(after gzip)

zensin•7/26/24, 1:20 AM

what about websockets make it a problem? I know it has binary mode

ChaikaOP•7/26/24, 1:22 AM

Depends on how you want to do it. Workers are short lived, part of the problem. If you wanted to actually run php/wp on workers establishing a websocket for each incoming request to another worker/durable object I think would be more expensive then just simple queries (espec if you can batch them into a few)

ChaikaOP•7/26/24, 1:23 AM

if you were going to do wp on an actual server you could use DOs to keep the websockets alive/sleep them, maybe more efficient to keep them open long term

ChaikaOP•7/26/24, 1:24 AM

sounds like a mess eitherway though lol, D1's cool for a lot of things, espec if we get read replicas, but WP is its own beast with the amount of queries it can make espec with plugins, not natively supporting sqlite even, and trying to run it is a whole different story

CChaika Depends on how you want to do it. Workers are short lived, part of the problem. ...

zensin•7/26/24, 1:31 AM

Wouldn't want to deal with added latency of API defeating much of the purpose.

A DO with hibernation seems okay as long as nothing about websockets makes it rly bad. Assuming the mysql lexer can relatively painlessly be ported to typescript or rust even, and then take some mysql proxy code (Of which there must be plenty) and slapt that in there. Better yet something like this would be an accelerated start running alongside wordpress (Or whatever): https://github.com/novnc/websockify

Zzensin Wouldn't want to deal with added latency of API defeating much of the purpose. ...

ChaikaOP•7/26/24, 1:35 AM

you're trying to run wp on a normal server, and use D1 as its database? Or what's your end goal, exactly?

CChaika you're trying to run wp on a normal server, and use D1 as its database? Or what'...

zensin•7/26/24, 1:35 AM

wp or anything else

zensin•7/26/24, 1:36 AM

I'm not claiming it would be good. just that it might be good

zensin•7/26/24, 1:36 AM

since the benefit (or one of them) of d1 is it's always close-by, right?

Zzensin since the benefit (or one of them) of d1 is it's always close-by, right?

ChaikaOP•7/26/24, 1:37 AM

as it is right now a D1 database is only in a singular location, no read replicas yet
and only where Durable Objects are supported https://where.durableobjects.live/

CChaika as it is right now a D1 database is only in a singular location, no read replica...

zensin•7/26/24, 1:37 AM

I was not aware there are no replicas

ChaikaOP•7/26/24, 1:38 AM

even when there is read replicas it's not going to be anywhere anywhere but it might make your idea more sensible

Original message was deleted

zensin•7/26/24, 1:38 AM

I addressed that above

Original message was deleted

ChaikaOP•7/26/24, 1:38 AM

well that's what he was talking about, translating the queries

ChaikaOP•7/26/24, 1:38 AM

idea could be cool espec with read replicas come out, just complex

Isaac McFadyen•7/26/24, 1:39 AM

Also:

take some mysql proxy code (of which there must be plenty)

There's likely plenty of proxy code, but almost certainly none currently that work on Workers.

Isaac McFadyen•7/26/24, 1:39 AM

Workers uses a custom implementation of a TCP Sockets API, not standard Node.JS APIs.

ChaikaOP•7/26/24, 1:39 AM

yeaaa would have to adapt to the web js environment, or hope there's enough node compat to make it work if it was just largely based on buffers and such, could potentially steal a good chunk of it

IIsaac McFadyen Also: > take some mysql proxy code (of which there must be plenty) There's like...

zensin•7/26/24, 1:39 AM

https://github.com/brettsmentek/Node-MySQL-to-SQLite
making no claims to how good or bad this is

GitHub

GitHub - brettsmentek/Node-MySQL-to-SQLite: Express app for convert...

Express app for converting MySQL data to SQLite. Contribute to brettsmentek/Node-MySQL-to-SQLite development by creating an account on GitHub.

Zzensin https://github.com/brettsmentek/Node-MySQL-to-SQLite making no claims to how goo...

Isaac McFadyen•7/26/24, 1:40 AM

Yes, and this would work with Node, but Workers aren't Node and can't run Express.

Original message was deleted

zensin•7/26/24, 1:40 AM

yes, rust can come too

IIsaac McFadyen Yes, and this would work with Node, but Workers aren't Node and can't run Expres...

zensin•7/26/24, 1:41 AM

I wouldn't use express. I'd use the mysql websocket proxy linked above

zensin•7/26/24, 1:41 AM

and obviously just take the part of the code from mysql-to-sqlite repo that does the translating

Original message was deleted

zensin•7/26/24, 1:45 AM

if I had to guess, this is probably the most advanced/worked on translator to sqlite: https://github.com/WordPress/sqlite-database-integration/pull/9 though idk if any of that wound up being wp-specific/not portable

Original message was deleted

zensin•7/26/24, 1:46 AM

sometimes 'because we can' is as important as 'because we should'

zensin•7/26/24, 1:55 AM

a rust mysql/sqlite lexer parser doesn't feel like a lot of overhead. websockets to a worker doesn't either but D1 not being replicated is a nonstarter. I will relent. For now, the dream is dashed.

Zzensin a rust mysql/sqlite lexer parser doesn't feel like a lot of overhead. websockets...

ChaikaOP•7/26/24, 1:57 AM

a lot of overhead vs not doing it and just having a local DB instead/replicating a local DB.
It might make sense for some simpler things once Read Replicas come out, but WP is a ton of queries, ehh

zensin•7/26/24, 1:58 AM

wait. I bet a DO could store an sqlite database as a blob, cache some queries, and replicate it across the network... lol

Zzensin wait. I bet a DO could store an sqlite database as a blob, cache some queries, a...

ChaikaOP•7/26/24, 1:59 AM

the original d1 implementation (now called alpha and going away) was using DO storage/all in DO afaik

ChaikaOP•7/26/24, 2:00 AM

I still have..a few

zensin•7/26/24, 2:00 AM

I have stuff to do too. Just like to waste some time before the boring parts. back to it I guess

Zzensin wait. I bet a DO could store an sqlite database as a blob, cache some queries, a...

ChaikaOP•7/26/24, 2:00 AM

DO's storage is pretty limited though, max value size is 128 KiB per key

Original message was deleted

zensin•7/26/24, 2:01 AM

yeah was about to say bring it in from object storage, which is a relatively dubious proposition

ChaikaOP•7/26/24, 2:02 AM

problem is R2's performance can be a bit weak and it's not replicated either

ChaikaOP•7/26/24, 2:02 AM

KV only has two central stores us/eu, and caching on every edge, so slow cold pulls

ChaikaOP•7/26/24, 2:02 AM

cached it's not bad

ChaikaOP•7/26/24, 2:02 AM

uncached..

ChaikaOP•7/26/24, 2:03 AM

If you really wanted a super fast cheap storage, embed your values straight into a worker and do a worker update when you need to update them, Workers max size is 10 MB on paid and if you only need infrequent updating.. lol

zensin•7/26/24, 2:04 AM

I could have sworn I once saw a repo that was doing some replication across DOs

Zzensin I could have sworn I once saw a repo that was doing some replication across DOs

ChaikaOP•7/26/24, 2:04 AM

maybe https://github.com/cloudflare/dog ?

zensin•7/26/24, 2:04 AM

that's it. not remembering all it did

ChaikaOP•7/26/24, 2:05 AM

hmm, should probably move to #workers-discussions or #off-topic since this isn't D1 related

zensin•7/26/24, 2:06 AM

right. anyway I can't get down that rabbit-hole or I won't get anything done. thanks for the going on about it with me

Phaze•7/26/24, 1:25 PM

Hi guys, i'm getting response times of >1 second when looking data up on D1, is this normal or am I doing something wrong?

Fabian•7/26/24, 2:03 PM

Hey, is there any known ETA yet for dynamic D1 bindings? :)

RexNormous•7/26/24, 2:07 PM

I'm trying to insert a potentially large amount of rows (up to 10,000 at max) at once to a D1 database. I'm getting failures due to maxing out the number of parameters to my sql statement.

I understand

batch

batch

is the solution to this, but can I send 10k sql inserts in one batch? Is there a limit to batch?

RRexNormous I'm trying to insert a potentially large amount of rows (up to 10,000 at max) at...

Hard@Work•7/26/24, 2:10 PM

The limit to a batch is just how many you can cram into memory. I would say batch is a decent bit more efficient than a single large insert

ChaikaOP•7/26/24, 2:10 PM

yea it was said before someone got 100k queries in a batch without issues: https://discord.com/channels/595317990191398933/992060581832032316/1180147342373044274

I use batch with upserts up to ~1-2k rows without issue

https://developers.cloudflare.com/workers/platform/limits/ free gets 1 mb max size, paid gets 10 mb

Similar Threads