I am considering creating a serverless database just for fun using AWS Lambda/@edge and S3/CDN to st

I am considering creating a serverless database just for fun using AWS Lambda/@edge and S3/CDN to store SQLite files.

Is this a good approach?

MMrinank I am considering creating a serverless database just for fun using AWS Lambda/@e...

fry69•10/28/24, 6:16 PM

How should this work? Do you want to write the whole SQLite file each time you do a SQL write operation?

Ffry69 How should this work? Do you want to write the whole SQLite file each time you d...

MrinankOP•10/28/24, 6:22 PM

yeah perform the required SQL operations in-memory, and then write the updated file back to S3 after each write operation, yeah i also know its not optimal for high-frequency write.
I got some interest from this video where he shared about hosting db on github pages:
https://www.youtube.com/watch?v=cYP0k_shdWc

YouTubeBufferhead

Host your Database for Free on Github Pages

Databases are an essential part of many modern web applications but running them can be really expensive. That’s why in this video I’ll show you how you can run an SQLite Database completely for free, on top of Girhub Pages. GitHub Pages allows you to host files and websites. But using the JavaScript library SQL.js-HTTPVFS it is possible to use ...

MMrinank I am considering creating a serverless database just for fun using AWS Lambda/@e...

Hello, I’m Allie!•10/28/24, 6:24 PM

I don't see why it can't work, but not sure why you are asking in #d1-database?

HHello, I’m Allie!I don't see why it can't work, but not sure why you are asking in #d1-database?

MrinankOP•10/28/24, 6:27 PM

I thought this channel might have the right people to ask. should I move this to the #off-topic ?

MMrinank I thought this channel might have the right people to ask. should I move this to...

fry69•10/28/24, 6:29 PM

That would be advisable, this channel is for questions/support for the D1 product from Cloudflare, not generic SQLite development.

Llambrospetrou Have you done the steps at https://developers.cloudflare.com/d1/build-with-d1/im...

Luiz Benevenuto•10/28/24, 6:51 PM

I have a D1 database and I want to duplicate to do some tests, not a local SQLIte

KaiM•10/29/24, 12:25 PM

Story time... I dove head-first into D1 for a new project of mine and while I've loved the DX (especially locally), I'm incredibly disappointed by the production performance. I'm located in Australia but I gave D1 a location hint of US East to try and keep latency as low as possible for US and European users. Now I'm trying to use my app from Australia in staging for the first time and it's unusable. No query has sub-600ms latency, even with Smart Placement enabled. These aren't complex queries either - selecting a single user based on the primary key column took over a second at its highest. Is there something I'm missing?

1984 Ford Laser•10/29/24, 12:39 PM

In what context are you using these queries? If you're just developing/testing, then does the specific speed matter too much? You could also create a different D1 db for each region, with a diff location hint. Then on the Worker that does the query, check the location of the requesting client and route to the nearest D1 db based on location

Original message was deleted

1984 Ford Laser•10/29/24, 12:41 PM

yeah this too

cybersentry•10/29/24, 12:46 PM

Can someone explain d1 - I’m trying to understand it. Can it work with geospatial data? I normally use Postgres and postgis

cybersentry•10/29/24, 12:48 PM

Can we work with geospatial data on it? I’m using Django so should I use SQLite locally?

Ccybersentry Can we work with geospatial data on it? I’m using Django so should I use SQLite ...

1984 Ford Laser•10/29/24, 12:49 PM

Can it be formatted as JSON or text? Then yes

cybersentry•10/29/24, 12:52 PM

I mean can’t anything be formatted as that? Just because it can doesn’t mean it should

Ccybersentry I mean can’t anything be formatted as that? Just because it can doesn’t mean it ...

1984 Ford Laser•10/29/24, 12:57 PM

We cant help you if we don't know what the data is. 'geospatial data' is so vague and broad, in theory basically any meaningful database solution could be used. A series of TXT documents formatted correctly could be used in theory

Original message was deleted

KaiM•10/29/24, 1:10 PM

I was under the impression that D1 had global replication. Does that exist at all?

KaiM•10/29/24, 1:17 PM

Oof, that sucks. At that point I’d rather just position my Worker in the same datacenter as the database and have the latency on the HTTP side. I gotta say, a database that is unable to be globally distributed but that’s designed for a compute platform that is global by default seems like a strange product offering.

KaiM•10/29/24, 1:18 PM

Doesn’t it try and find a middle ground? Because I’m still getting 500+ms latency on D1 queries with smart placement enabled.

KaiM•10/29/24, 1:21 PM

Yeah I saw those. Most of my endpoints do a few queries, most with at least one batch. As expected the first is consistently the slowest, but I still got higher latency than I would expect for subsequent ones. I’ll profile it all again tomorrow.

TomK•10/29/24, 2:05 PM

I've encountered a significant number of "D1_ERROR: internal error" messages today. we haven't updated our code. Has there been any changes in D1 today?

TTomK I've encountered a significant number of "D1_ERROR: internal error" messages tod...

fry69•10/29/24, 2:08 PM

In doubt check -> status there have been issues a few hours ago with pages, but not D1, so IDK (though it also showed "internal error").

fry69•10/29/24, 2:09 PM

"internal error" can literally mean anything.

fry69•10/29/24, 2:11 PM

Unless we see more reports here or elsewhere, it might also be something with your database/setup which Cloudflare can't handle for some reason.

TomK•10/29/24, 2:12 PM

The error message has now changed to "D1_ERROR: D1 storage operation exceeded timeout which caused object to be reset."

Ffry69 Unless we see more reports here or elsewhere, it might also be something with yo...

TomK•10/29/24, 2:12 PM

Okay, thank you for your response. I will check further.

nato•10/29/24, 2:35 PM

how can I make 2 different cloudflare workers apps use same D1 db in local dev?

nato•10/29/24, 2:35 PM

I see it uses ./wrangler/state/v3/d1 but since both projects are on seperate folders

nato•10/29/24, 2:36 PM

is there any way to specify path?

nato•10/29/24, 2:37 PM

nvm it was mentioned in docs

npx wrangler dev --persist-to <DIRECTORY>

npx wrangler dev --persist-to <DIRECTORY>

TTomK Okay, thank you for your response. I will check further.

Stan Angeloff•10/29/24, 3:01 PM

I'm another customer that's observing lots of random D1 failures today (and in general a lot more since September, lots of lost connection, internal error, etc.) Just thought I'd chime in so you know you are not alone.

sz19•10/29/24, 4:01 PM

It started at 12PM UTC, I've been receiving these errors since then.
D1 storage operation exceeded timeout which caused object to be reset. / Network connection lost. / internal error

Ssz19 It started at 12PM UTC, I've been receiving these errors since then. D1 storage ...

TomK•10/29/24, 4:38 PM

I also encountered the same issue at the same time.

TTomK I also encountered the same issue at the same time.

sz19•10/29/24, 4:49 PM

is your d1 database located somewhere in asia?

Ssz19 is your d1 database located somewhere in asia?

TomK•10/29/24, 4:57 PM

yes

TomK•10/29/24, 4:59 PM

I'm not sure if this has anything to do with the maintenance program in the Singapore region?

TomK•10/29/24, 5:02 PM

TTomK Click to see attachment

fry69•10/29/24, 5:09 PM

That is still in the future if I read this correctly (UTC)

fry69•10/29/24, 5:10 PM

(should start in <1h)

fry69•10/29/24, 5:12 PM

The hiccups could of course be due to preparations for this maintenance or other things.

Ffry69 That is still in the future if I read this correctly (UTC)

TomK•10/29/24, 5:14 PM

Yes, it was future time, which is a very far-fetched reason indeed, but other than that I can't associate it with any other reason, since the code wasn't updated.:NotLikeThis:

sz19•10/29/24, 5:38 PM

Update: errors seems to be gone now on mine

sz19•10/29/24, 5:44 PM

There was about a solid 5 mins giving me this error:
Error: D1_ERROR: Internal error while starting up D1 storage caused object to be reset.
, then after that i no longer receive d1 errors. Hopefully it stays like this

rubenf•10/29/24, 8:41 PM

Our D1 database has been unavailable for the last 10mins withL: D1_ERROR: Internal error while starting up D1 storage caused object to be reset.D1_ERROR: Internal error while starting up D1 storage caused object to be reset.

rubenf•10/29/24, 8:42 PM

even the cloud console can't access it

rubenf•10/29/24, 8:42 PM

Are D1 databases beta/unfit for prod?