Cloudflare Developers•17mo ago

rn i have some csv data converted to json... how would i automatically upload them to d1?

rn i have some csv data converted to json...

how would i automatically upload them to d1?

Zzorous rn i have some csv data converted to json... how would i automatically upload t...

Hurby•8/29/24, 6:10 PM

I think you can just write basic script on cf workers to upload them

itsmatteomanf•8/29/24, 7:02 PM

Text...?

itsmatteomanf•8/29/24, 7:06 PM

There is none in SQLite, they differ in other database types.

itsmatteomanf•8/29/24, 7:06 PM

https://www.sqlite.org/datatype3.html

itsmatteomanf•8/29/24, 7:10 PM

Not as far as I know, the limits are the ones state on the dev docs.

itsmatteomanf•8/29/24, 7:10 PM

And those of SQLite, of course

IEatBeans•8/30/24, 2:09 AM

Is CF aware of the random intermittent D1 Errors that cause a 500? Such as:

Internal error in D1 storage caused object to be reset

IIEatBeans Is CF aware of the random intermittent D1 Errors that cause a 500? Such as: In...

Max (@rozenmd)•8/30/24, 5:47 AM

we are, and have rolled out a bug fix for it

on the off chance this is something new, what's your database id?

MMax (@rozenmd)we are, and have rolled out a bug fix for it on the off chance this is somethin...

IEatBeans•8/30/24, 5:49 AM

That’s great to hear. Here’s the ID if needed:

e2e1eb93-f9ba-4615-aea7-bb44f89b61c3

o7•8/30/24, 6:02 AM

Getting D1_ERROR: Internal error in D1 storage caused object to be reset.

Oo7 Getting D1_ERROR: Internal error in D1 storage caused object to be reset.

Max (@rozenmd)•8/30/24, 6:10 AM

what's your database id?

MMax (@rozenmd)what's your database id?

o7•8/30/24, 6:10 AM

b2ea5e97-258f-4752-a68b-4dea4b10bf6c

o7•8/30/24, 6:11 AM

its calmed down now but it was intermittent. alongside 520s on a different site which didnt belong to me

o7•8/30/24, 6:18 AM

it might've been 85951405-f007-4d17-b1d4-094ed462f582 as well, which is the auth layer, but it would let me authenticate before returning that error when trying to access the 2nd DB pasted above

RexNormous•8/31/24, 1:42 AM

Also seeing intermittent Internal error in D1 storage caused object to be reset.Internal error in D1 storage caused object to be reset.
Database ID: 82459739-eaf4-4ad4-ac50-24ff60338cb1

Original message was deleted

Vikash•8/31/24, 2:07 AM

Vikash•8/31/24, 2:10 AM

I had to convert my data files to JSON > Upload to R2 and run a scheduled task to import all the files one by one using db.batch command

cosbgn•8/31/24, 8:09 AM

D1_ERROR: Internal error in D1 storage caused object to be reset.

Any ideas what could cause this?

Hiky•8/31/24, 2:02 PM

what is the problem with this the log shows client is undefined pls help

Alex Taccard•8/31/24, 4:55 PM

When one runs npx wrangler d1 createnpx wrangler d1 create to create a D1 database, he has to manually insert the new D1 binding configuration to the file

wrangler.toml

wrangler.toml

. However, if we execute this command in a folder that already contains

wrangler.toml

wrangler.toml

, can the binding insertion be achieved automatically, e.g. by a command option? If not, wouldn't this be a nice feature to implement?

Original message was deleted

Max (@rozenmd)•8/31/24, 7:14 PM

will take a look on Monday

Original message was deleted

Matt Silverlock•8/31/24, 9:55 PM

You can use (e.g.) https://grafana.com/grafana/plugins/fifemon-graphql-datasource/ with our GraphQL API

Grafana Labs

GraphQL Data Source plugin for Grafana | Grafana Labs

GraphQL Data Source

lifeiscontent•8/31/24, 10:27 PM

If I have a worker and a pages app operating on the same d1 database will it result in data corruption?

Llifeiscontent If I have a worker and a pages app operating on the same d1 database will it res...

Chaika•8/31/24, 10:35 PM

no, can have as many bound to it as you want/use api/dash, etc. Your D1 DB runs in a single location on a single machine

R in Tokyo•9/1/24, 12:08 AM

To the D1 user may I have some clarification on whether D1 is globally distributed? I’ve read that there are read only clones replicated close to your workers?

IIsaac McFadyen See this message for some past details from an employee - in short, maybe a bit ...

R in Tokyo•9/1/24, 12:11 AM

Why is there cloudflare marketing material mentioned “read replication” if they don’t have it?

Mozzy•9/1/24, 12:11 AM

+1 on the increase of Internal error in D1 storage caused object to be reset.Internal error in D1 storage caused object to be reset. errors

R in Tokyo•9/1/24, 12:15 AM

How to make data global?

D1 GA has focused on reliability and developer experience. Now, we plan on extending D1 to better support globally-distributed applications.

In the Workers model, an incoming request invokes serverless execution in the closest data center. A Worker application can scale globally with user requests. Application data, however, remains stored in centralized databases, and global user traffic must account for access round trips to data locations. For example, a D1 database today resides in a single location

So it’s obviously not global?

R in Tokyo•9/1/24, 12:17 AM

Can somone here decipher the marketing material here ?

https://blog.cloudflare.com/building-d1-a-global-database/

The Cloudflare Blog

Building D1: a Global Database

D1, Cloudflare’s SQL database, is now generally available. With new support for 10GB databases, data export, and enhanced query debugging, we empower developers to build production-ready applications with D1 to meet all their relational SQL needs

R in Tokyo•9/1/24, 12:20 AM

This is literally a screenshot of replicated D1

https://cf-assets.www.cloudflare.com/slt3lc6tev37/2R1p29j20c7szuRmY2Sjlp/d3da1494232913b556806d624a0c8f14/image3-34.png

R in Tokyo•9/1/24, 12:30 AM

Ok so D1 is actually not global. Dear Cloudflare transparency goes a long way, and improves customer loyalty.

This stunts will have your customers jump to the nearest honest competitor.

It’s 2024, that article should be title “D1 is not global yet”

RR in Tokyo Ok so D1 is actually not global. Dear Cloudflare transparency goes a long way, a...

James•9/1/24, 12:35 PM

Agreed. This is a question I've had to answer time and time again because of the confusing marketing. Sorry. Example: https://www.reddit.com/r/CloudFlare/comments/1c2a7im/is_d1_rest_api_performant_and_production_ready/kz8u3i5/

Kiwi•9/1/24, 10:35 PM

Is Smart Placement a likely cause of the D1_ERROR increase I'm seeing?

Original message was deleted

Cyb3r-Jak3•9/1/24, 10:37 PM

I'm confused. You want to query a D1 database from Grafana?

veeque•9/2/24, 4:03 PM

without deleting the db and making another one, how can i make the db empty as if i just made it? (basically a drop)

ceddy•9/2/24, 4:15 PM

Is D1 scalable for large applications ? how does it compare to postgres

Cceddy Is D1 scalable for large applications ? how does it compare to postgres

useless•9/2/24, 4:21 PM

I want to ask the same question, but compared with RDS PostgreSQL

Uuseless I want to ask the same question, but compared with RDS PostgreSQL

ceddy•9/2/24, 4:23 PM

https://tenor.com/view/calculating-puzzled-math-confused-confused-look-gif-14677181

Tenor

Uuseless I want to ask the same question, but compared with RDS PostgreSQL

ceddy•9/2/24, 4:23 PM

but that's the same bro

useless•9/2/24, 4:23 PM

hahaha

Uuseless hahaha

ceddy•9/2/24, 4:24 PM

https://tenor.com/view/%D9%81%D9%87%D8%AF-what-huh-stare-confused-gif-16855622

Tenor

RR in Tokyo Can somone here decipher the marketing material here ? https://blog.cloudflare....

Vy•9/2/24, 6:07 PM

This blog post states that read replication is not yet available and was intended to share an early preview of our Sessions API for user feedback. Neither D1 docs or product website claim that global replication is available - we are opening up the private beta soon.

VVy This blog post states that read replication is not yet available and was intende...

James•9/2/24, 6:12 PM

The "global" language, old blogs, and marketing material at https://www.cloudflare.com/en-gb/developer-platform/d1/ for example that state offers asynchronous read replicationoffers asynchronous read replication definitely do imply that, and might need tightening up a bit. Confusion around this comes up a lot.

JJames The "global" language, old blogs, and marketing material at https://www.cloudfla...

Vy•9/2/24, 6:14 PM

Thanks for catching the async piece, I'll get that corrected. would also note that "global" has many connotations here at Cloudflare so we try to be explicit if we mean global replication

Original message was deleted

Vy•9/2/24, 6:16 PM

What is the concern here? The 50K limit is not a hard limit. We communicate a number because often "unlimited" raises its own questions

VVy Thanks for catching the async piece, I'll get that corrected. would also note th...

James•9/2/24, 6:37 PM

I hear you about "global replication" vs "global", but I'm not sure most people share that same conclusion Cloudflare does. Is my Postgres database I have hosted on some random server a "global database" just because I can access it globally? It's a little meaningless in terms of marketing, and I completely understand why it confuses people.

João Castro•9/2/24, 7:26 PM

Are there any news about supporting transactions in D1? Transactions are fundamental for building reliable applications, and even though D1 is in GA, we don't have this functionality.

JJoão Castro Are there any news about supporting transactions in D1? Transactions are fundame...

Vy•9/2/24, 7:32 PM

explicit transactions are not in development right now. D1's batch() API does execute a SQL transaction - https://developers.cloudflare.com/d1/build-with-d1/d1-client-api/#batch-statements

Cloudflare Docs

Query D1 | Cloudflare D1 docs

D1 is compatible with most SQLite’s SQL convention since it leverages SQLite’s query engine. D1 client API allows you to interact with a D1 database from within a Worker.

rn i have some csv data converted to json... how would i automatically upload them to d1?

Similar Threads

Similar Threads

Similar Threads