Like with the new DO SQLite storage?

HHello, I’m Allie!Like with the new DO SQLite storage?

ac•10/23/24, 5:51 PM

Not very familiar with DOs, but looking at the docs/syntax, probably yes? Mainly a way to have a generic D1 binding and a D1.getByIdD1.getById function

Aac Not very familiar with DOs, but looking at the docs/syntax, probably yes? Mainly...

Hello, I’m Allie!OP•10/23/24, 5:54 PM

Yeah, that sounds right, though note that a DO by default doesn't expose the SQL interface to the calling Worker

ac•10/23/24, 6:10 PM

Yeah, I'm not using Durable Objects, just D1, so just wondering if this functionality is truly planned for D1 (and maybe coming soon?)

Sheraz•10/23/24, 8:28 PM

Anyone else getting "D1_ERROR: D1 is overloaded. Too many requests queued" more than usual today?

Aac Are there any plans to make it possible to have dynamic service bindings for D1?...

John Spurlock•10/23/24, 11:21 PM

yep, it is on their roadmap

https://blog.cloudflare.com/making-full-stack-easier-d1-ga-hyperdrive-queues/

Jadugar_Jaggu•10/25/24, 6:16 AM

Facing perfromance issues with D1 today, taking some time to retrive data

anand248•10/26/24, 3:00 AM

wrangler d1 migrations apply remote failing to find pending migrations to apply, ends without applying anything.
wrangler.toml

[[d1_databases]]
binding = "DB"
database_name = "name-db"
database_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
preview_database_id = "LOCAL-MY-DB" # Required for Pages local development
migrations_dir = "migrations"

[vars]

[[env.production.d1_databases]]
binding = "DB"
database_name = "name-db"
database_id = "8someguid63a"
migrations_dir = "migrations"

[[d1_databases]]
binding = "DB"
database_name = "name-db"
database_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
preview_database_id = "LOCAL-MY-DB" # Required for Pages local development
migrations_dir = "migrations"

[vars]

[[env.production.d1_databases]]
binding = "DB"
database_name = "name-db"
database_id = "8someguid63a"
migrations_dir = "migrations"

My migrations migrations folder do have some unapplied .sql.sql files, but it fails to find them

 ~/tess-fs/web/ [main] npx wrangler d1 migrations list name-db

wrangler 3.83.0
-------------------

No migrations to apply!
 ~/tess-fs/web/ [main] npx wrangler d1 migrations list name-db --remote

wrangler 3.83.0
-------------------

 ~/tess-fs/web/ [main*]
-------------------

Aanand248 wrangler d1 migrations apply remote failing to find pending migrations to apply,...

anand248•10/26/24, 3:12 AM

found the solution, I am suppose to use the actual database_iddatabase_id of remote database in [[d1_databases]][[d1_databases]] section. I thought, when we supply --remote parater, it picks the [[env.production.d1_databases]][[env.production.d1_databases]] section. I am struggling to understand wrangler.tomlwrangler.toml file structure.

fry69•10/26/24, 9:11 PM

May be better placed here -> #what-i-built

nullptr•10/26/24, 10:42 PM

Is there a way to mirror prod d1 to local? All the examples i've seen online were doing migrations.

Shinebayar G•10/27/24, 5:21 AM

Hello. I'm getting this error when running npx drizzle-kit pushnpx drizzle-kit push command. Anyone know what's going on?

[✓] Pulling schema from database...
Error: 7500: not authorized: SQLITE_AUTH

[✓] Pulling schema from database...
Error: 7500: not authorized: SQLITE_AUTH

Shinebayar G•10/27/24, 5:22 AM

API token has edit permission on D1 resource.

Edit: It seems something is broken on Drizzle Kit.

Ssilentdevnull Does D1 support the FTS5 Extension of SQLite? I'm trying to figure out if D1 can...

fry69•10/27/24, 9:34 PM

Looks like it does? -> https://community.cloudflare.com/t/d1-support-for-virtual-tables/607277/3?u=fry69

Ssilentdevnull Does D1 support the FTS5 Extension of SQLite? I'm trying to figure out if D1 can...

lambrospetrou•10/28/24, 11:25 AM

Yes: https://developers.cloudflare.com/d1/reference/sql-statements/#sqlite-extensions

ItsWendell•10/28/24, 1:30 PM

Did anyone else had any unexplained reporting in billable usage for D1? On the 25th of October it's saying it suddenly had 1 million row reads, though it's not verifiable through the D1 analytics page, see thread for screenshots:

Luiz Benevenuto•10/28/24, 4:23 PM

Hello, how can i duplicate a D1 database ?

LLuiz Benevenuto Hello, how can i duplicate a D1 database ?

lambrospetrou•10/28/24, 4:33 PM

Not out of the box. You need to export the data and import it in the new database.

Llambrospetrou Not out of the box. You need to export the data and import it in the new databas...

Luiz Benevenuto•10/28/24, 5:02 PM

Yeah but I'm having issues doing that

Luiz Benevenuto•10/28/24, 5:02 PM

All the time

LLuiz Benevenuto Yeah but I'm having issues doing that

lambrospetrou•10/28/24, 5:49 PM

Have you done the steps at https://developers.cloudflare.com/d1/build-with-d1/import-export-data/#convert-sqlite-database-files to properly prepare your import file?

Mrinank•10/28/24, 6:13 PM

I am considering creating a serverless database just for fun using AWS Lambda/@edge and S3/CDN to store SQLite files.

Is this a good approach?

MMrinank I am considering creating a serverless database just for fun using AWS Lambda/@e...

fry69•10/28/24, 6:16 PM

How should this work? Do you want to write the whole SQLite file each time you do a SQL write operation?

Ffry69 How should this work? Do you want to write the whole SQLite file each time you d...

Mrinank•10/28/24, 6:22 PM

yeah perform the required SQL operations in-memory, and then write the updated file back to S3 after each write operation, yeah i also know its not optimal for high-frequency write.
I got some interest from this video where he shared about hosting db on github pages:
https://www.youtube.com/watch?v=cYP0k_shdWc

YouTubeBufferhead

Host your Database for Free on Github Pages

Databases are an essential part of many modern web applications but running them can be really expensive. That’s why in this video I’ll show you how you can run an SQLite Database completely for free, on top of Girhub Pages. GitHub Pages allows you to host files and websites. But using the JavaScript library SQL.js-HTTPVFS it is possible to use ...

MMrinank I am considering creating a serverless database just for fun using AWS Lambda/@e...

Hello, I’m Allie!OP•10/28/24, 6:24 PM

I don't see why it can't work, but not sure why you are asking in #d1-database?

HHello, I’m Allie!I don't see why it can't work, but not sure why you are asking in #d1-database?

Mrinank•10/28/24, 6:27 PM

I thought this channel might have the right people to ask. should I move this to the #off-topic ?

MMrinank I thought this channel might have the right people to ask. should I move this to...

fry69•10/28/24, 6:29 PM

That would be advisable, this channel is for questions/support for the D1 product from Cloudflare, not generic SQLite development.

Llambrospetrou Have you done the steps at https://developers.cloudflare.com/d1/build-with-d1/im...

Luiz Benevenuto•10/28/24, 6:51 PM

I have a D1 database and I want to duplicate to do some tests, not a local SQLIte

KaiM•10/29/24, 12:25 PM

Story time... I dove head-first into D1 for a new project of mine and while I've loved the DX (especially locally), I'm incredibly disappointed by the production performance. I'm located in Australia but I gave D1 a location hint of US East to try and keep latency as low as possible for US and European users. Now I'm trying to use my app from Australia in staging for the first time and it's unusable. No query has sub-600ms latency, even with Smart Placement enabled. These aren't complex queries either - selecting a single user based on the primary key column took over a second at its highest. Is there something I'm missing?

1984 Ford Laser•10/29/24, 12:39 PM

In what context are you using these queries? If you're just developing/testing, then does the specific speed matter too much? You could also create a different D1 db for each region, with a diff location hint. Then on the Worker that does the query, check the location of the requesting client and route to the nearest D1 db based on location

Original message was deleted

1984 Ford Laser•10/29/24, 12:41 PM

yeah this too

cybersentry•10/29/24, 12:46 PM

Can someone explain d1 - I’m trying to understand it. Can it work with geospatial data? I normally use Postgres and postgis

cybersentry•10/29/24, 12:48 PM

Can we work with geospatial data on it? I’m using Django so should I use SQLite locally?

Ccybersentry Can we work with geospatial data on it? I’m using Django so should I use SQLite ...

1984 Ford Laser•10/29/24, 12:49 PM

Can it be formatted as JSON or text? Then yes

cybersentry•10/29/24, 12:52 PM

I mean can’t anything be formatted as that? Just because it can doesn’t mean it should

Ccybersentry I mean can’t anything be formatted as that? Just because it can doesn’t mean it ...

1984 Ford Laser•10/29/24, 12:57 PM

We cant help you if we don't know what the data is. 'geospatial data' is so vague and broad, in theory basically any meaningful database solution could be used. A series of TXT documents formatted correctly could be used in theory

Original message was deleted

KaiM•10/29/24, 1:10 PM

I was under the impression that D1 had global replication. Does that exist at all?

KaiM•10/29/24, 1:17 PM

Oof, that sucks. At that point I’d rather just position my Worker in the same datacenter as the database and have the latency on the HTTP side. I gotta say, a database that is unable to be globally distributed but that’s designed for a compute platform that is global by default seems like a strange product offering.

KaiM•10/29/24, 1:18 PM

Doesn’t it try and find a middle ground? Because I’m still getting 500+ms latency on D1 queries with smart placement enabled.

KaiM•10/29/24, 1:21 PM

Yeah I saw those. Most of my endpoints do a few queries, most with at least one batch. As expected the first is consistently the slowest, but I still got higher latency than I would expect for subsequent ones. I’ll profile it all again tomorrow.

TomK•10/29/24, 2:05 PM

I've encountered a significant number of "D1_ERROR: internal error" messages today. we haven't updated our code. Has there been any changes in D1 today?

TTomK I've encountered a significant number of "D1_ERROR: internal error" messages tod...

fry69•10/29/24, 2:08 PM

In doubt check -> status there have been issues a few hours ago with pages, but not D1, so IDK (though it also showed "internal error").

Like with the new DO SQLite storage?

Similar Threads

Similar Threads

Similar Threads