Cloudflare Developers•3y ago

What's the best practices for an ETL to export data from D1?

Ffranz101 What's the best practices for an ETL to export data from D1?

Dahoom•9/30/23, 11:37 AM

For the beta I'm currenly using community member nora's repo https://github.com/nora-soderlund/cloudflare-d1-backups

GitHub

GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a ...

A package to dump a copy of your experimental D1 database through a worker. - GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a copy of your experimental D1 database through a worker.

Dahoom•9/30/23, 11:38 AM

it works pretty well but I dont think there is any official system the beta system yet

franz101OP•9/30/23, 11:44 AM

Ah interesting I saw the dump function. I was curious as I was wondering about a local python connection to query the db. I guess you would need a worker that evals the query. I'm testing d1 for my ai news aggregator:
https://ml-news-liard.vercel.app/

Ffranz101 Ah interesting I saw the dump function. I was curious as I was wondering about a...

Dahoom•9/30/23, 11:49 AM

You probably need a worker yeah, I know for a fact you can connect directly to turso currently with libsql

Jason Hostetter•9/30/23, 3:25 PM

Is it correct that there is no way to programmatically create or connect to different databases for users without an enterprise account?

JJason Hostetter Is it correct that there is no way to programmatically create or connect to diff...

Cyb3r-Jak3•9/30/23, 3:30 PM

I'm pretty sure even with enterprise you can't

Jason Hostetter•9/30/23, 3:32 PM

There was mention of creating one db per client in the beta announcement, but only for enterprise

Cyb3r-Jak3•9/30/23, 3:32 PM

Oh workers for platforms

Jason Hostetter•9/30/23, 3:32 PM

Although the description wasn’t super clear to me

Jason Hostetter•9/30/23, 3:32 PM

Yeah

Cyb3r-Jak3•9/30/23, 3:33 PM

Yeah, sorry haven't used WfP but that will allow you to

Jason Hostetter•9/30/23, 3:34 PM

Is workers for platforms enterprise only?

Cyb3r-Jak3•9/30/23, 3:35 PM

It is

Jason Hostetter•9/30/23, 3:35 PM

Bahhh

Jason Hostetter•9/30/23, 3:38 PM

Ok well, maybe I’ll have to try to hack something together with turso or something

JJason Hostetter Is it correct that there is no way to programmatically create or connect to diff...

Matt Silverlock•9/30/23, 4:01 PM

You can programmatically create databases - the D1 API isn’t limited to enterprise accounts at all. There is a limit on the # of DBs you create (at the moment) on regular accounts but we will increase this very soon

Jason Hostetter•9/30/23, 4:14 PM

Oh nice! Can I dynamically bind a database depending on some parameter or key passed in the request to the worker?

JJason Hostetter Oh nice! Can I dynamically bind a database depending on some parameter or key pa...

Matt Silverlock•9/30/23, 4:17 PM

No, that is something we don’t yet support. Working on it. If you can bind any resource in your account it would break the security model (how do you bind only the right things?) - so the work is focused on APIs to help group/tag resources vs “any”

Jason Hostetter•9/30/23, 4:37 PM

Great to hear, looking forward to it. Was worried that functionality would be locked behind enterprise, which would effectively kill my project (at least with D1)

Samantas5855•9/30/23, 10:00 PM

Can I add data to a d1 database from workers and a python script?

Samantas5855•9/30/23, 10:04 PM

also does delete count as a write or is it free?

SSamantas5855 Can I add data to a d1 database from workers and a python script?

Cyb3r-Jak3•9/30/23, 10:19 PM

You'd would have the use the API to add the content from a python script

Cyb3r-Jak3•9/30/23, 10:22 PM

And I'd imagine that deleting rows counts as a write but not about if you delete a D1 database entirely

ajgeiss0702•10/1/23, 12:58 AM

Once d1 has read replication, any idea on how consistent it would be? I’m currently planning a feature that would require semi-strong consistency
(Please ping me if you reply)

Aakira.eth One thing I missed is the dynamic creation of databases. Every user that creates...

Matt Silverlock•10/1/23, 5:31 AM

d1-database

Aajgeiss0702 Once d1 has read replication, any idea on how consistent it would be? I’m curren...

Unsmart•10/1/23, 5:32 AM

What does "semi-strong consistency" mean its either strongly consistent or its not

I'd assume read replication would be eventually consistent

Sameer Ali•10/1/23, 7:46 AM

You can share .wrangler folder between multiple workers and point to same migration directory from your wrangler.toml

UUnsmart What does "semi-strong consistency" mean its either strongly consistent or its n...

ajgeiss0702•10/1/23, 12:19 PM

By semi-strong I meant under 5 seconds

UUnsmart What does "semi-strong consistency" mean its either strongly consistent or its n...

Matt Silverlock•10/1/23, 2:31 PM

Both terms are very binary. D1 will have causal (session) consistency - “read your own writes” - with read replication

UUnsmart What does "semi-strong consistency" mean its either strongly consistent or its n...

Matt Silverlock•10/1/23, 2:32 PM

https://jepsen.io/consistency

tadeumaia•10/1/23, 2:39 PM

With the release of hyperdrive, does it mean that today I could have a "read replica" of my data at all Cloudflare point of presence at my country, while with D1 I'll have to wait for the release of the read replicas? (being my country is Brazil, i don't even see a hint of D1 for south america)

Ttadeumaia With the release of hyperdrive, does it mean that today I could have a "read rep...

Matt Silverlock•10/1/23, 5:43 PM

Hyperdrive doesn’t create replicas: it caches reads. Big difference.

MMatt Silverlock Hyperdrive doesn’t create replicas: it caches reads. Big difference.

tadeumaia•10/1/23, 5:59 PM

I tried to imply that with the quotes, sorry for the miscommunication, and thanks for your answer. I’m looking for a pure performance standpoint. Looking to rewrite some parts of a product that I have for the maximum read performance. Right now all located on AWS SA and cloudflare on front caching the html static. Users all over Brazil, would love to have the data distributed for when the cache bursts. But I was actually reading hyperdrive doc further and the cache looks short for my use case as I don’t have that much reads because all is cached statically.

Ttadeumaia I tried to imply that with the quotes, sorry for the miscommunication, and thank...

Matt Silverlock•10/1/23, 6:01 PM

Got it. Hyperdrive is still going to be a very effective intermediate cache - on top of the connection pooling & perf. Without it, cache misses will likely hurt a lot in terms of latency.

Ttadeumaia With the release of hyperdrive, does it mean that today I could have a "read rep...

Matt Silverlock•10/1/23, 6:02 PM

wrt D1 in South America - we’re working on it.

MMatt Silverlock wrt D1 in South America - we’re working on it.

tadeumaia•10/1/23, 6:16 PM

Thanks. That’s great to know. Rewriting the stack from .net 4 to workers after the TCP connection announcement. Most of it is ready, was going with a hosted AWS Postgres for development but looking for the feasibility at the hyperdrive/D1 after this week’s announcement. Will hyperdrive work with D1 or will D1 have that builtin with replicas at all the pop’s?

Ttadeumaia Thanks. That’s great to know. Rewriting the stack from .net 4 to workers after t...

Matt Silverlock•10/1/23, 6:37 PM

Separate things. Hyperdrive for databases not on Cloudflare. D1 as D1. Search this channel for my posts on “read replication” for D1

mukhtharcm•10/2/23, 1:54 AM

is downloading the database coming back soon?

Mmukhtharcm is downloading the database coming back soon?

mukhtharcm•10/2/23, 1:54 AM

We had this ability to download backup, but since time machine came, we can't

Mmukhtharcm We had this ability to download backup, but since time machine came, we can't

Dahoom•10/2/23, 4:03 AM

https://github.com/nora-soderlund/cloudflare-d1-backups

GitHub

GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a ...

DDahoom https://github.com/nora-soderlund/cloudflare-d1-backups

mukhtharcm•10/2/23, 4:17 AM

Ty!

Mmukhtharcm Ty!

Dahoom•10/2/23, 4:18 AM

dont thank me, thank nora. Shes the one who made it

Original message was deleted

mukhtharcm•10/2/23, 6:15 AM

Thank you @nora for this!
I've been searching forever for something like this!

AAlexander I'm guessing you have a production database of some kind and are you planning to...

Laurent Pellegrino•10/2/23, 12:54 PM

Thanks for your message.

Yep, it's somewhat like a read-only replica. But the challenge is that about 90% of the data is refreshed daily, so diff computing or event-based reactions won't be of much help. I think @rozenmd's generic response might be due to Cloudflare not wanting backups to be abused. If I'm not mistaken, backup imports don't count as writes. So, in theory, someone could set up a local database, fill it up, back it up, and then import to Cloudflare D1 at no cost. Those backups weren't just for disaster recovery; they fit my scenario perfectly. But with only Timetravel remaining, the $1/M write cost is steep for those of us needing to import large databases daily or even weekly only.

Max (@rozenmd)•10/2/23, 12:59 PM

backup imports don't count as writes

D1's beta backend doesn't have backup imports - you can write to the DB, and you're billed as a write.

Metrics do not appear for D1's older alpha dbs because the metrics feature was built only for the beta backend.

What's the best practices for an ETL to export data from D1?

Similar Threads

Similar Threads

Similar Threads