Cloudflare Developers•3y ago

Does D1 support (or will) streams?

SSameer Ali Does D1 support (or will) streams?

Unsmart•8/30/23, 6:42 PM

d1-database

Sameer AliOP•8/30/23, 6:45 PM

I think that is different question than mine. I am trying to see if i can stream the response from SELECT query directly to client instead of loading the result in worker memory. let's say i have a query that returns large amount of data, which is over the worker memory limit of 128MB, streaming the response without loading into buffer would be only way to workaround worker memory limit.

SSameer Ali I think that is different question than mine. I am trying to see if i can stream...

Max (@rozenmd)•8/30/23, 7:08 PM

That's another thing we're looking at

Max (@rozenmd)•8/30/23, 7:09 PM

but main priority is d1-database

Sameer AliOP•8/30/23, 7:12 PM

Thanks. It is must thing to have for many use cases. But it can wait.

itsmatteomanf•8/30/23, 8:17 PM

People, use indexes OMEGALUL

itsmatteomanf•8/30/23, 8:17 PM

itsmatteomanf•8/30/23, 8:21 PM

Yeah, they do, but given my volume of reads... it makes sense to improve them.

itsmatteomanf•8/30/23, 8:23 PM

Yeah, of course it's a tradeoff. One is required to be an index, the other might not make sense.

Unsmart•8/30/23, 8:25 PM

Using a NoSQL db it would be 1 write 1 delete and most auto delete a ttl

UUnsmart Using a NoSQL db it would be 1 write 1 delete and most auto delete a ttl <:EZ:74...

itsmatteomanf•8/30/23, 8:26 PM

But it wouldn't work well for my use case. I like NoSQL, but it's not always the best.

Unsmart•8/30/23, 8:26 PM

I wasnt even talking about your usecase cuz idk what it is but Leos would

Unsmart•8/30/23, 8:28 PM

If DOs had TTL on keys it would be better priced than kv but you would have to set an alarm so there it would still be multiple writes in weird ways lol

Original message was deleted

itsmatteomanf•8/30/23, 8:31 PM

Yeah, SQL is nice for those things.

Unsmart•8/30/23, 8:32 PM

Yeah KV has ttl so you wouldnt need to cron and actually afaik those deletes are free

itsmatteomanf•8/30/23, 8:33 PM

You could honestly avoid the expire index, tbh. If it takes a few milliseconds more in a cron it's totally fine.

itsmatteomanf•8/30/23, 8:33 PM

D1 is actually impressively fast.

Original message was deleted

Matt Silverlock•8/30/23, 9:06 PM

At those rates will you ever exceed the included read & write usage in the $5/mo plan? Feels like the wrong kind of optimization.

schwartzmj•8/31/23, 12:29 AM

Anybody have some links to a blog post or github repo that sets up SvelteKit with D1 (using Wrangler?) for both local dev and prod? I can't seem to find good material anywhere and would love to test out D1 and some other CF offerings, but the local dev situation seems a bit complicated

Sschwartzmj Anybody have some links to a blog post or github repo that sets up SvelteKit wit...

Cyb3r-Jak3•8/31/23, 12:45 AM

https://developers.cloudflare.com/d1/examples/d1-and-sveltekit/
https://github.com/elithrar/svelte-demo-d1

CCyb3r-Jak3 https://developers.cloudflare.com/d1/examples/d1-and-sveltekit/ https://github.c...

schwartzmj•8/31/23, 12:47 AM

At a glance, it looks like these only work in production. I can't seem to find examples that include the local dev setup / workflow. Apologies if I missed anything in either of those examples from my phone here. Plan to research and look more when I get back

Sschwartzmj At a glance, it looks like these only work in production. I can't seem to find e...

Cyb3r-Jak3•8/31/23, 12:49 AM

The docs are in production but you basially use the pages dev command locally with a D1 binding and it will run locally

Sschwartzmj At a glance, it looks like these only work in production. I can't seem to find e...

Dahoom•8/31/23, 12:58 AM

The only way it can work locally is if you build it after each new change running npm run buildnpm run build and then running wrangler pages dev .svelte-kit/cloudflarewrangler pages dev .svelte-kit/cloudflare

Dahoom•8/31/23, 12:58 AM

Make sure you have included your d1 binding on wrangler.toml so it can use the local d1 you have

DDahoom The only way it can work locally is if you build it after each new change runnin...

schwartzmj•8/31/23, 1:00 AM

Does this cause issues with HMR? I'll test this out in a bit

Dahoom•8/31/23, 1:01 AM

It's not possible with HMR

Dahoom•8/31/23, 1:01 AM

That's the issue

Dahoom•8/31/23, 1:01 AM

I'll give you my npm command you can use after each new change though

Dahoom•8/31/23, 1:03 AM

in package.json I have this for scripts:

        "scripts": { 
                 "dev": "wrangler pages dev .svelte-kit/cloudflare --local", 
                 "live": "vite build -w", 
                 "build": "run.bat & vite build",

        "scripts": { 
                 "dev": "wrangler pages dev .svelte-kit/cloudflare --local", 
                 "live": "vite build -w", 
                 "build": "run.bat & vite build",

Dahoom•8/31/23, 1:04 AM

This is the contents of run.bat:

rmdir /s /q .svelte-kit\cloudflare

rmdir /s /q .svelte-kit\cloudflare

Dahoom•8/31/23, 1:05 AM

run.bat is used as a workaround and sometimes I had an issue where the cloudflare build file is not fully cleared each time I wanted to run the pages local build

DDahoom run.bat is used as a workaround and sometimes I had an issue where the cloudflar...

schwartzmj•8/31/23, 1:05 AM

Ok thank you! I'll try this all out. Really appreciate it

Dahoom•8/31/23, 1:07 AM

no problem, @ me with a question if you had any because there isn't much documentation regarding this aspect and I had to work most of it out myself

addison•8/31/23, 2:12 AM

Hey, does D1 have FTS now? I haven't checked it out since open alpha launched

Aaddison Hey, does D1 have FTS now? I haven't checked it out since open alpha launched

Dahoom•8/31/23, 3:02 AM

What's FTS?

DDahoom What's FTS?

addison•8/31/23, 3:15 AM

Full Text Search

Aaddison Hey, does D1 have FTS now? I haven't checked it out since open alpha launched

Max (@rozenmd)•8/31/23, 5:54 AM

yes

riskiidrus•8/31/23, 8:19 AM

Hi, I wanted to ask about the reliability of D1 in case I ever need to add a new column to a large table.

Let's assume I have a "users" table containing 1 million rows. One time I wanted to add a new column to the table. Is the process as fast as I imagined?

Although I have never experienced something like that before, I will definitely experience it someday. And it's probably not just me.

Rriskiidrus Hi, I wanted to ask about the reliability of D1 in case I ever need to add a new...

itsmatteomanf•8/31/23, 8:22 AM

Adding a column, which is empty at the start, is very fast on any SQL database.

itsmatteomanf•8/31/23, 8:22 AM

You'll then need to do up to a million writes to fill it.

riskiidrus•8/31/23, 8:29 AM

Thank you for your answer. I was very relieved to hear that.

But when can we use D1 for production? I'm really looking forward to it

Elliot Hesp•8/31/23, 9:30 AM

Does anyone know how you're supposed to run Drizzle Kit migrations on D1? https://orm.drizzle.team/kit-docs/quick

Elliot Hesp•8/31/23, 9:31 AM

No issues making the migration, but I'm not sure how it's going to connect to the DB outside of the wrangler / fetch handler

Elliot Hesp•8/31/23, 9:40 AM

Got it: https://sat0shi.dev/posts/drizzle-migration/

Type-Safely database migration on Cloudflare D1 with Drizzle ORM

Drizzle Kit allows you to easily migrate your database schema on Cloudflare D1.

itsmatteomanf•8/31/23, 10:39 AM

I'm not sure there is a way, but can you get the keys from a JSON object within D1? I know I can get the values, but not sure about the keys.

Iitsmatteomanf I'm not sure there is a way, but can you get the keys from a JSON object within ...

Isaac McFadyen•8/31/23, 10:40 AM

https://stackoverflow.com/questions/70759212/select-distinct-keys-from-a-json-column-in-sqlite not sure if D1 exposes them specifically but SQLite's JSON1 allows for it.

Stack Overflow

select distinct keys from a json column in SQLite

a TEXT column of a SQLite database contains json values like these one:
[{"key1":"116"},{"key2":"260"},{"key3":"123"},{"key4":&...

itsmatteomanf•8/31/23, 10:40 AM

Oh, that looks promising.

Matt Silverlock•8/31/23, 11:25 AM

json_eachjson_each - https://developers.cloudflare.com/d1/learning/querying-json/#supported-functions

Query JSON · Cloudflare D1 docs

D1 has built-in support for querying and parsing JSON data stored within a database. This enables you to:

MMatt Silverlock `json_each` - https://developers.cloudflare.com/d1/learning/querying-json/#suppo...

itsmatteomanf•8/31/23, 11:43 AM

Yeah, that one above worked

I didn’t think it returned the key, too. It works, it’s not the fastest but it’s 10s of keys on tens of thousands of rows

itsmatteomanf•8/31/23, 11:44 AM

@rozenmd adding to the request for indexes, maybe show views, too

Does D1 support (or will) streams?

Similar Threads

Similar Threads

Similar Threads