Is there any way to interleave two tables on disk? I have a table that has a 1 to many relationship

Is there any way to interleave two tables on disk? I have a table that has a 1 to many relationship with another table. I'm never going to query the "many" table to ask if it connects to the "one" table, always the other way around. I'd like to tell Cloudflare that they can optimize it by keeping the records from the 1 table next to the corresponding records from the many table by interleaving the rows on disk. That way it only needs to pull 1 disk block up when I query for data. Is this possible?

ShroomOP•12/18/23, 4:06 PM

Also, any way to bump up the 2 GB limit? Ikr it's not production ready, but I'd like to use it for a purely non-profit idea and I'm not sure if I'll stay under that limit

SShroom Is there any way to interleave two tables on disk? I have a table that has a 1 t...

kenton•12/18/23, 5:37 PM

D1 is built on SQLite, so this is only possible if SQLite itself contains such a feature. I don't think it does. But at the same time, I suspect this kind of feature would not have so much benefit given we're running SQLite backed by a local SSD. It'd be more important for a distributed database that might split data across many machines, but D1 doesn't do that.

ShroomOP•12/18/23, 8:16 PM

It doesn't do that? I thought the big advantage of d1 is that it's present at all your cdns. Does it just replicate the whole db everywhere?

SShroom It doesn't do that? I thought the big advantage of d1 is that it's present at al...

Isaac McFadyen•12/18/23, 8:21 PM

D1 is currently single location (either close to where you are when you create your DB, or in a geographic area of your choosing selected when you create the DB).

Isaac McFadyen•12/18/23, 8:22 PM

In the future there will be read replication support. But for now it's single location.

kian•12/18/23, 8:25 PM

Smart Placement will always run your Worker near the D1 DB

Original message was deleted

Max (@rozenmd)•12/18/23, 8:29 PM

I would use integers

IIsaac McFadyen D1 is currently single location (either close to where you are when you create y...

ShroomOP•12/18/23, 8:30 PM

Good to know. Most of my data can also be modeled as key-value pairs so I'll use that + hyperdrive.

Kkian Smart Placement will always run your Worker near the D1 DB

Isaac McFadyen•12/18/23, 8:43 PM

Note that Smart Placement places the Worker near the D1 DB. You'll still have to talk to the Worker back in Canada (in your example).

zegevlier•12/18/23, 9:26 PM

I would recommend ensuring R2 and D1 are in the same region, then enabeling smart placement.

Zzegevlier I would recommend ensuring R2 and D1 are in the same region, then enabeling smar...

Max (@rozenmd)•12/18/23, 10:09 PM

Smart placement should automagically do that for you (as in decide the optimal place to put the Worker)

zegevlier•12/18/23, 10:12 PM

It wouldn't put R2 and D1 in the same place, right?

SShroom It doesn't do that? I thought the big advantage of d1 is that it's present at al...

kenton•12/19/23, 2:11 AM

We are working on replication for D1, but even when it is replicated, each replica will store the entire database, so there's no need for a query to gather data from multiple machines. We might later add support for sharding, which puts different data on different machines, but we'll likely do this based on an explicit shard key provided by the application, so you'll be able to control which data lives together using shard keys rather than by interleaving tables.

Unsmart•12/19/23, 4:23 AM

I dont even want to know what the cost of D1 would be if it was every datacenter LUL

IIsaac McFadyen In the future there will be read replication support. But for now it's single lo...

riskiidrus•12/19/23, 7:06 AM

I created a database and chose a location in asia pacific. But why is the latency still so high? It even took up to 1 minute just to view the home page of my web application.

I am in Indonesia and My Web Application targets Users from Southeast Asian Countries.

I have selected the right location for My Database but the Latency is still so High. I don't know where in the Asia Pacific Region the D1 Data Center is located.

Kkenton We are working on replication for D1, but even when it is replicated, each repli...

huw•12/19/23, 7:26 AM

+1 for jurisdictional control over those shards

MMustafa Akın Since D1 supports 50k DBs now, but I want to employ a per-tenant database, in th...

GuillaumeLakano•12/19/23, 8:53 AM

The temporary solution is to PATCH the settings of your worker to attach new bindings ( REST API ) and then you can uses D1 through without the REST API

GGuillaumeLakano The temporary solution is to PATCH the settings of your worker to attach new bin...

Matt Silverlock•12/19/23, 12:10 PM

These are still bindings.

MMatt Silverlock These are still bindings.

GuillaumeLakano•12/19/23, 12:25 PM

Indeed, I've misread the question of @Mustafa Akın
Can you share with us any news about your future solutions?

GGuillaumeLakano Indeed, I've misread the question of @Mustafa Akın Can you share with us any ne...

Matt Silverlock•12/19/23, 12:28 PM

Not yet. We’ve shared a lot here already - it’s something we’re working on and will be part of/ahead of GA.

kian•12/19/23, 8:37 PM

D1 is SQLite, and doesn't really have the concept of users that you'd apply RLS to anyways.

kian•12/19/23, 8:45 PM

I'm not sure what you mean by that.

kian•12/19/23, 8:45 PM

You use the REST API or bind a Worker to the database.

Original message was deleted

Max (@rozenmd)•12/19/23, 9:17 PM

it's up to you to add application logic to show data based on authentication/authorization

Original message was deleted

Max (@rozenmd)•12/19/23, 9:19 PM

I recommend you go through the getting started guide to understand how it works: https://developers.cloudflare.com/d1/get-started/

Get started · Cloudflare D1 docs

This guide will instruct you through:

MMax (@rozenmd)You've pretty much identified all the sharp edges with working with multiple dat...

huw•12/19/23, 11:31 PM

(FYI for the discussion above this is how you PATCH extra bindings into a worker—cheers Max!)

mw10013•12/20/23, 1:38 AM

Is there a way to "reset" a database (ie. get it into the same state as if just created). This would be helpful with prototyping where I would like to reset the database and run the single migration I have that I'm editing. Currently, I'm deleting the database, creating another a new one with the same name, binding it to the pages app, updating the database id in the wrangler.toml, and finally applying the edited migration. Would be great to simply reset the database and then apply the edited migration. With local dev, I delete the

.wrangler

.wrangler

directory. Longing for a way to reset databases in the cloud

Max (@rozenmd)•12/20/23, 9:53 AM

I just throw a

DROP TABLE IF EXISTS...

DROP TABLE IF EXISTS...

at the start of my migrations

Cy•12/20/23, 11:07 AM

For anyone that’s used D1, what’s the performance look like?

Cy•12/20/23, 11:07 AM

I’m using turso and planetscale right now and I’m thinking about making the switch

CCy For anyone that’s used D1, what’s the performance look like?

JustinNoel•12/20/23, 1:17 PM

I was using Planetscale and switched to D1. much lower latency (improved about 50-100ms) for most areas I tested against.

Faster DB queries too but planetscale wAs pretty fast. Sorry, can't recall numbers.

Cy•12/20/23, 1:31 PM

Yeah I’m thinking latency will be the biggest thing given that I usually use pages

Cy•12/20/23, 1:32 PM

But I’m just worried about the query time tbh

CCy But I’m just worried about the query time tbh

Max (@rozenmd)•12/20/23, 1:34 PM

query time is fast with an index, getting data out rapidly is still a

//TODO

//TODO

NNico Hey, you've probably been asked this question many times, but I couldn't find an...

Chaika•12/20/23, 6:24 PM

D1 DBs are only accessible via bound workers or via the API/API Tokens you make. The only Workers that can access them are ones which are bound. IP Restrictions would be something you could do in the worker/firewall level

Chaika•12/20/23, 6:25 PM

It's not like they're normal SQL DBs that you can connect to externally, D1 is based on SQLite, no connection strings

Vy•12/20/23, 9:22 PM

@Nico What type of access controls are you interested in from a bound Worker itself?

Andrew Murphy•12/20/23, 10:55 PM

Is there a way for me to export my entire D1 to a file/something so I can put it into reporting tools like PowerBI etc?

Andrew Murphy•12/20/23, 10:57 PM

wrangler d1 backup create xxx

wrangler d1 backup create xxx

gives me the error

Experimental DBs do not support backups yet

Experimental DBs do not support backups yet

blakelockbrown•12/20/23, 11:09 PM

Is D1 storage limit really 2GB? How is anyone supposed to run a production system on this?

Bblakelockbrown Is D1 storage limit really 2GB? How is anyone supposed to run a production syste...

Cyb3r-Jak3•12/20/23, 11:12 PM

You’re not yet. It’s still beta

AAndrew Murphy Is there a way for me to export my entire D1 to a file/something so I can put it...

Cyb3r-Jak3•12/20/23, 11:13 PM

d1-database

Mmax Hi all, I'm wondering if someone can help me debug why my D1 latency is so high,...

mukhtharcm•12/21/23, 3:44 AM

did you got any solution?

kian•12/21/23, 3:50 AM

Your D1 database is located in a single location, so there are two things to look at.

1) Look at the returned

meta

meta

object from your query, it will include the duration that D1 actually took to execute your query.

2) Look at where your D1 database is located (

wrangler d1 info <database_name>

wrangler d1 info <database_name>

) - if it's in a different region to you, and you're making multiple calls, it'll be pretty slow due to the network round-trip time.

Victor•12/21/23, 12:37 PM

Hi, guys, is there any docs to explain how many column types in d1?

VVictor Hi, guys, is there any docs to explain how many column types in d1?

GuillaumeLakano•12/21/23, 12:56 PM

D1 uses SQLite, so the documentation is here: https://www.sqlitetutorial.net/sqlite-data-types/

GGuillaumeLakano D1 uses SQLite, so the documentation is here: https://www.sqlitetutorial.net/sql...

Victor•12/21/23, 12:57 PM

Max (@rozenmd)•12/21/23, 1:11 PM

the official docs are decent too: https://www.sqlite.org/datatype3.html