Does a backup restore (not using time traval) incurs a charge (i.e. does it rows as writes)?

LLaurent Pellegrino Does a backup restore (not using time traval) incurs a charge (i.e. does it rows...

The backup commands are from a time when it was necessary to do backups with D1. Time travel lets you rollback commit by commit, so it's not as useful.

Laurent PellegrinoOP•9/29/23, 9:14 PM

@rozenmd I am asking because of the use case I described just above.

Laurent PellegrinoOP•9/29/23, 9:16 PM

I'd be willing to shell out $0.1 per million writes for my situation, but $1 per million writes is certainly off-putting.

Old Man Maker•9/30/23, 2:02 AM

Hey, can someone help me understand the D1 insert SQL a little better please.

I have an object def that looks like this:

interface DetailsDataType{
    description:string,
    ref:string //Primary Key in Table
}

interface DetailsDataType{
    description:string,
    ref:string //Primary Key in Table
}

I'm getting an array of these objects that I want to insert into my D1.

I get how to inject a single string, but the object array is not happening for me.
I assume there is some shorthand way of doing this, don't want to put an ugly for loop in there.

This is the INSERT code, where the bind statement is incorrect because descData is an array:

descData: DetailsDataType[] = <ARRAY OF OBJECTS>;

const { results } = await c.env.DATABASE.prepare("INSERT INTO details (ref, description) VALUES (?, ?) RETURNING *")
        .bind(descData.description, descData.ref)
        .run()

const { results } = await c.env.DATABASE.prepare("INSERT INTO details (ref, description) VALUES (?, ?) RETURNING *")
        .bind(descData.description, descData.ref)
        .run()

Can this even be done this way?

Old Man Maker•9/30/23, 2:24 AM

I'm assuming I will need to use the Batching statement?

Old Man Maker•9/30/23, 2:42 AM

Ok - just got this to work with a looping batcher. Pretty sweet.

Can I make a suggestion please - if the use of hyphens '-' in names is causing so many issues, why not just blanket ban the use in any named object.

It just took me 10 mins to spot the issue with the table name, when it should have been rejected as a valid table name at the point of creation. Perhaps the wrangler tool does this, but the dashboard does not.

Thoughts?

franz101•9/30/23, 11:34 AM

What's the best practices for an ETL to export data from D1?

Ffranz101 What's the best practices for an ETL to export data from D1?

Dahoom•9/30/23, 11:37 AM

For the beta I'm currenly using community member nora's repo https://github.com/nora-soderlund/cloudflare-d1-backups

GitHub

GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a ...

A package to dump a copy of your experimental D1 database through a worker. - GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a copy of your experimental D1 database through a worker.

Dahoom•9/30/23, 11:38 AM

it works pretty well but I dont think there is any official system the beta system yet

franz101•9/30/23, 11:44 AM

Ah interesting I saw the dump function. I was curious as I was wondering about a local python connection to query the db. I guess you would need a worker that evals the query. I'm testing d1 for my ai news aggregator:
https://ml-news-liard.vercel.app/

Ffranz101 Ah interesting I saw the dump function. I was curious as I was wondering about a...

Dahoom•9/30/23, 11:49 AM

You probably need a worker yeah, I know for a fact you can connect directly to turso currently with libsql

Jason Hostetter•9/30/23, 3:25 PM

Is it correct that there is no way to programmatically create or connect to different databases for users without an enterprise account?

JJason Hostetter Is it correct that there is no way to programmatically create or connect to diff...

Cyb3r-Jak3•9/30/23, 3:30 PM

I'm pretty sure even with enterprise you can't

Jason Hostetter•9/30/23, 3:32 PM

There was mention of creating one db per client in the beta announcement, but only for enterprise

Cyb3r-Jak3•9/30/23, 3:32 PM

Oh workers for platforms

Jason Hostetter•9/30/23, 3:32 PM

Although the description wasn’t super clear to me

Jason Hostetter•9/30/23, 3:32 PM

Yeah

Cyb3r-Jak3•9/30/23, 3:33 PM

Yeah, sorry haven't used WfP but that will allow you to

Jason Hostetter•9/30/23, 3:34 PM

Is workers for platforms enterprise only?

Cyb3r-Jak3•9/30/23, 3:35 PM

It is

Jason Hostetter•9/30/23, 3:35 PM

Bahhh

Jason Hostetter•9/30/23, 3:38 PM

Ok well, maybe I’ll have to try to hack something together with turso or something

JJason Hostetter Is it correct that there is no way to programmatically create or connect to diff...

Matt Silverlock•9/30/23, 4:01 PM

You can programmatically create databases - the D1 API isn’t limited to enterprise accounts at all. There is a limit on the # of DBs you create (at the moment) on regular accounts but we will increase this very soon

Jason Hostetter•9/30/23, 4:14 PM

Oh nice! Can I dynamically bind a database depending on some parameter or key passed in the request to the worker?

JJason Hostetter Oh nice! Can I dynamically bind a database depending on some parameter or key pa...

Matt Silverlock•9/30/23, 4:17 PM

No, that is something we don’t yet support. Working on it. If you can bind any resource in your account it would break the security model (how do you bind only the right things?) - so the work is focused on APIs to help group/tag resources vs “any”

Jason Hostetter•9/30/23, 4:37 PM

Great to hear, looking forward to it. Was worried that functionality would be locked behind enterprise, which would effectively kill my project (at least with D1)

Samantas5855•9/30/23, 10:00 PM

Can I add data to a d1 database from workers and a python script?

Samantas5855•9/30/23, 10:04 PM

also does delete count as a write or is it free?

SSamantas5855 Can I add data to a d1 database from workers and a python script?

Cyb3r-Jak3•9/30/23, 10:19 PM

You'd would have the use the API to add the content from a python script

Cyb3r-Jak3•9/30/23, 10:22 PM

And I'd imagine that deleting rows counts as a write but not about if you delete a D1 database entirely

ajgeiss0702•10/1/23, 12:58 AM

Once d1 has read replication, any idea on how consistent it would be? I’m currently planning a feature that would require semi-strong consistency
(Please ping me if you reply)

Aakira.eth One thing I missed is the dynamic creation of databases. Every user that creates...

Matt Silverlock•10/1/23, 5:31 AM

d1-database

Aajgeiss0702 Once d1 has read replication, any idea on how consistent it would be? I’m curren...

Unsmart•10/1/23, 5:32 AM

What does "semi-strong consistency" mean its either strongly consistent or its not

I'd assume read replication would be eventually consistent

Sameer Ali•10/1/23, 7:46 AM

You can share .wrangler folder between multiple workers and point to same migration directory from your wrangler.toml

UUnsmart What does "semi-strong consistency" mean its either strongly consistent or its n...

ajgeiss0702•10/1/23, 12:19 PM

By semi-strong I meant under 5 seconds

UUnsmart What does "semi-strong consistency" mean its either strongly consistent or its n...

Matt Silverlock•10/1/23, 2:31 PM

Both terms are very binary. D1 will have causal (session) consistency - “read your own writes” - with read replication

UUnsmart What does "semi-strong consistency" mean its either strongly consistent or its n...

Matt Silverlock•10/1/23, 2:32 PM

https://jepsen.io/consistency

tadeumaia•10/1/23, 2:39 PM

With the release of hyperdrive, does it mean that today I could have a "read replica" of my data at all Cloudflare point of presence at my country, while with D1 I'll have to wait for the release of the read replicas? (being my country is Brazil, i don't even see a hint of D1 for south america)

Ttadeumaia With the release of hyperdrive, does it mean that today I could have a "read rep...

Matt Silverlock•10/1/23, 5:43 PM

Hyperdrive doesn’t create replicas: it caches reads. Big difference.

MMatt Silverlock Hyperdrive doesn’t create replicas: it caches reads. Big difference.

tadeumaia•10/1/23, 5:59 PM

I tried to imply that with the quotes, sorry for the miscommunication, and thanks for your answer. I’m looking for a pure performance standpoint. Looking to rewrite some parts of a product that I have for the maximum read performance. Right now all located on AWS SA and cloudflare on front caching the html static. Users all over Brazil, would love to have the data distributed for when the cache bursts. But I was actually reading hyperdrive doc further and the cache looks short for my use case as I don’t have that much reads because all is cached statically.

Ttadeumaia I tried to imply that with the quotes, sorry for the miscommunication, and thank...

Matt Silverlock•10/1/23, 6:01 PM

Got it. Hyperdrive is still going to be a very effective intermediate cache - on top of the connection pooling & perf. Without it, cache misses will likely hurt a lot in terms of latency.

Ttadeumaia With the release of hyperdrive, does it mean that today I could have a "read rep...

Matt Silverlock•10/1/23, 6:02 PM

wrt D1 in South America - we’re working on it.

MMatt Silverlock wrt D1 in South America - we’re working on it.

tadeumaia•10/1/23, 6:16 PM

Thanks. That’s great to know. Rewriting the stack from .net 4 to workers after the TCP connection announcement. Most of it is ready, was going with a hosted AWS Postgres for development but looking for the feasibility at the hyperdrive/D1 after this week’s announcement. Will hyperdrive work with D1 or will D1 have that builtin with replicas at all the pop’s?

Ttadeumaia Thanks. That’s great to know. Rewriting the stack from .net 4 to workers after t...

Matt Silverlock•10/1/23, 6:37 PM

Separate things. Hyperdrive for databases not on Cloudflare. D1 as D1. Search this channel for my posts on “read replication” for D1

Does a backup restore (not using time traval) incurs a charge (i.e. does it rows as writes)?

Similar Threads

Similar Threads

Similar Threads