I rely heavily on the [d1 http api](https://developers.cloudflare.com/api/operations/cloudflare-d1-q

I rely heavily on the d1 http api in my postgres app. I am trying to use d1 as an edge cache to postgres so that cloudflare workers can connect directly to d1 instead of postgres reducing latency. In order to do that I added an insert trigger in postgres to call the

d1 http api

d1 http api

. Here is the issue that I am facing
D1 api sometimes fails randomly with no clear reason. I have implemented a queue system to collect all of the failed calls and here are some error msgs I get

{"content":"{\"success\":false,\"errors\":[{\"code\":10000,\"message\":\"Authentication error\"}]}\n","error_msg":null,"status_code":500}

{"content":"{\"success\":false,\"errors\":[{\"code\":10000,\"message\":\"Authentication error\"}]}\n","error_msg":null,"status_code":500}

(Strange, as I do not change authentication and only some calls fail with this error

{"content":"{\n  \"result\": null,\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 7501,\n      \"message\": \"Internal error: D1 API returned: (503 Service Unavailable) [truncate html, see screenshot]

{"content":"{\n  \"result\": null,\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 7501,\n      \"message\": \"Internal error: D1 API returned: (503 Service Unavailable) [truncate html, see screenshot]

{"content":"{\n  \"result\": null,\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 7501,\n      \"message\": \"Internal error: D1 API returned: (503 Service Unavailable) [truncate html, see screenshot]

{"content":"{\n  \"result\": null,\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 7501,\n      \"message\": \"Internal error: D1 API returned: (503 Service Unavailable) [truncate html, see screenshot]

{"content":"{\n  \"result\": null,\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 7502,\n      \"message\": \"Internal error: db error: FATAL: server conn crashed?\"\n    }\n  ],\n  \"messages\": []\n}","error_msg":null,"status_code":500}

{"content":"{\n  \"result\": null,\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 7502,\n      \"message\": \"Internal error: db error: FATAL: server conn crashed?\"\n    }\n  ],\n  \"messages\": []\n}","error_msg":null,"status_code":500}

{"content":"{\n  \"result\": null,\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 7502,\n      \"message\": \"Internal error: db error: FATAL: server conn crashed?\"\n    }\n  ],\n  \"messages\": []\n}","error_msg":null,"status_code":500}

{"content":"{\n  \"result\": null,\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 7502,\n      \"message\": \"Internal error: db error: FATAL: server conn crashed?\"\n    }\n  ],\n  \"messages\": []\n}","error_msg":null,"status_code":500}

{"content":"{\n  \"result\": [],\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 7500,\n      \"message\": \"Cannot resolve D1 due to transient issue on remote node.\"\n    }\n  ],\n  \"messages\": []\n}","error_msg":null,"status_code":500}

{"content":"{\n  \"result\": [],\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 7500,\n      \"message\": \"Cannot resolve D1 due to transient issue on remote node.\"\n    }\n  ],\n  \"messages\": []\n}","error_msg":null,"status_code":500}

{"content":"{\n  \"result\": [],\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 7500,\n      \"message\": \"Cannot resolve D1 due to transient issue on remote node.\"\n    }\n  ],\n  \"messages\": []\n}","error_msg":null,"status_code":500}

{"content":"{\n  \"result\": [],\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 7500,\n      \"message\": \"Cannot resolve D1 due to transient issue on remote node.\"\n    }\n  ],\n  \"messages\": []\n}","error_msg":null,"status_code":500}

{"content":"{\n  \"result\": null,\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 7500,\n      \"message\": \"Internal error: Flipper failed: Flipper API returned: error sending request for url (https://flipper.sd.cfplat.com/features/edgeworker?acct=8640544): error trying to connect: dns error: failed to lookup address information: Name or service not known\"\n    }\n  ],\n  \"messages\": []\n}","error_msg":null,"status_code":500}

{"content":"{\n  \"result\": null,\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 7500,\n      \"message\": \"Internal error: Flipper failed: Flipper API returned: error sending request for url (https://flipper.sd.cfplat.com/features/edgeworker?acct=8640544): error trying to connect: dns error: failed to lookup address information: Name or service not known\"\n    }\n  ],\n  \"messages\": []\n}","error_msg":null,"status_code":500}

{"content":"{\n  \"result\": null,\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 7500,\n      \"message\": \"Internal error: Flipper failed: Flipper API returned: error sending request for url (https://flipper.sd.cfplat.com/features/edgeworker?acct=8640544): error trying to connect: dns error: failed to lookup address information: Name or service not known\"\n    }\n  ],\n  \"messages\": []\n}","error_msg":null,"status_code":500}

{"content":"{\n  \"result\": null,\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 7500,\n      \"message\": \"Internal error: Flipper failed: Flipper API returned: error sending request for url (https://flipper.sd.cfplat.com/features/edgeworker?acct=8640544): error trying to connect: dns error: failed to lookup address information: Name or service not known\"\n    }\n  ],\n  \"messages\": []\n}","error_msg":null,"status_code":500}

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

WcaleNieWolnyOP•1/24/24, 9:21 AM

`{"content":null,"error_msg":"Timeout was reached","status_code":null}` - this is my app timing out the HTTP request. I am not sure how long the timeout is, but it's likely around 5+ seconds.
- {"content":"upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection termination","error_msg":null,"status_code":503}{"content":"upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection termination","error_msg":null,"status_code":503}
- {"content":"{\n \"result\": null,\n \"success\": false,\n \"errors\": [\n {\n \"code\": 7501,\n \"message\": \"Internal error: D1 API returned: (409 Conflict) error code: 1018\"\n }\n ],\n \"messages\": []\n}","error_msg":null,"status_code":500}{"content":"{\n \"result\": null,\n \"success\": false,\n \"errors\": [\n {\n \"code\": 7501,\n \"message\": \"Internal error: D1 API returned: (409 Conflict) error code: 1018\"\n }\n ],\n \"messages\": []\n}","error_msg":null,"status_code":500}
- {"content":"<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n","error_msg":null,"status_code":502}{"content":"<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n","error_msg":null,"status_code":502}
Given that there were A LOT of requests into D1 (around 150K since 18th january) and that I only ever got random 200 errors like that I do not want to say that d1 is bad. However aside from retyring the failed requests is there is anything else I could try? I can easily retry d1 failed requests but I really do not want to resort to that.

WWcaleNieWolny I rely heavily on the [d1 http api](https://developers.cloudflare.com/api/operat...

kenton•1/24/24, 3:19 PM

The HTTP API at api.cloudflare.com is intended to be used for configuration and debugging, not live serving. It won't have anywhere near the availability and performance of the D1 API called from a Worker. If you want HTTP access to your D1 database that can be used as part of a production stack, I would recommend writing a Worker that implements an HTTP API backed by a D1 binding, rather than using api.cloudflare.com.

Kkenton The HTTP API at api.cloudflare.com is intended to be used for configuration and ...

Max (@rozenmd)•1/24/24, 3:25 PM

for example: https://github.com/elithrar/http-api-d1-example

Will•1/25/24, 12:40 AM

Let’s say I made 90 database, each database is to hold one day worth of data, this is to circumvent the database size limit

Then I bind all of them to my worker

How can I tell which db to use to my worker?

Let’s say the if input is 2024-01-25 then use env.db_2024-01-25

I am struggling to use user input as variable name

villainy•1/25/24, 11:13 AM

is there a timeline for d1? when could we expect a stable release?

villainy•1/25/24, 11:13 AM

i want to use it in prod

villainy•1/25/24, 11:15 AM

ik know its a beta, but would like to see how it stacks up against another sqlite service, like Turso

Foreignity•1/25/24, 11:45 AM

Hello everyone,

I'm trying to deploy a worker with a D1 DB and I'm getting this error: (what am I doing wrong?)

kian•1/25/24, 11:45 AM

You need to use Module Workers (

export default

export default

) rather than Service Workers (

addEventListener

addEventListener

) with D1.

Kkian You need to use Module Workers (`export default`) rather than Service Workers (`...

Foreignity•1/25/24, 11:48 AM

Thank you @kian

Vvillainy is there a timeline for d1? when could we expect a stable release?

Max (@rozenmd)•1/25/24, 12:48 PM

https://blog.cloudflare.com/d1-open-beta-is-here/

We’re setting a clear goal: we want to make D1 “generally available” (GA) for production use-cases by early next year (Q1 2024)

The end of Q1 is April, so around then

Original message was deleted

Max (@rozenmd)•1/25/24, 6:31 PM

A fix for that is in-progress by @geelen

yeehow•1/26/24, 12:22 AM

Is there a way to tell wrangler where to store the db locally?

I have a monorepo with CF pages and a worker both using the same DB.
Using miniflare on the pages app to do this. I'm guessing I should just do the same for the worker?

Yyeehow Is there a way to tell wrangler where to store the db locally? I have a monorep...

Benedikt•1/26/24, 12:55 AM

You're looking for "--persist-to" I believe

BBenedikt You're looking for "--persist-to" I believe

yeehow•1/26/24, 12:59 AM

Aha! Totally missed that, thanks!

thiccvitalik•1/26/24, 1:09 AM

This just made me monkey tilt. It's not even listed on the d1 limits page.

sqlite3_limit(db, SQLITE_LIMIT_COMPOUND_SELECT, 5);

sqlite3_limit(db, SQLITE_LIMIT_COMPOUND_SELECT, 5);

The SQLite default is 500 btw.

EDIT: just found out SQLite has window functions

Tthiccvitalik This just made me monkey tilt. It's not even listed on the d1 limits page. ```cp...

kian•1/26/24, 2:01 AM

3 is the recommended high-security value by SQLite.

kian•1/26/24, 2:01 AM

D1 aims for https://www.sqlite.org/security.html over the defaults.

mian.m.arslan•1/26/24, 10:16 AM

Hi Everyone
Can anyone help me to connect D1 database to mySql workbench?

Mmian.m.arslan Hi Everyone Can anyone help me to connect D1 database to mySql workbench?

Max (@rozenmd)•1/26/24, 10:23 AM

D1 is SQLite over HTTP, not MySQL - it might work with tools that support custom JDBC drivers: https://github.com/isaac-mcfadyen/d1-jdbc-driver

Max (@rozenmd)•1/26/24, 1:49 PM

https://github.com/isaac-mcfadyen/d1-jdbc-driver/blob/main/src/main/java/org/isaacmcfadyen/D1Queryable.java#L80 surprised this endpoint hasn't changed in 2 years

spookily_•1/26/24, 3:07 PM

quick question: do timestamptimestamp / datetimedatetime fields work in UTC?

spookily_•1/26/24, 3:07 PM

eg if i do something like...

insert into table (expires_at) values (datetime('now', '+1 hour'));

insert into table (expires_at) values (datetime('now', '+1 hour'));

will that give the same result regardless of the TZ the worker node is on?

Sspookily_quick question: do `timestamp` / `datetime` fields work in UTC?

Matt Silverlock•1/26/24, 3:12 PM

They default to UTC

Matt Silverlock•1/26/24, 3:13 PM

https://www.sqlite.org/lang_datefunc.html

spookily_•1/26/24, 3:14 PM

awesome, thank you, figured as much just wanted to confirm

Original message was deleted

Isaac McFadyen•1/26/24, 3:20 PM

lol yes.

Isaac McFadyen•1/26/24, 3:20 PM

I kinda gave up on that project since I was busy and didn't have much time to maintain.

PPaul Shriner Basically what I am trying to do is take a comma delimited list of sorted number...

Chinoman10•1/26/24, 4:22 PM

Funny, that's how vector databases work

jeremy•1/26/24, 4:42 PM

Hey, I don't know if someone can help me understand how best to integrate D1 with NextJS. I am confused between whether I should be using a separate Worker, or if I should be using a binding directly in the Page (web app).

WcaleNieWolnyOP•1/26/24, 6:13 PM

Does D1 live in the edge or is it run centrally?

Martin•1/26/24, 6:31 PM

@nora do we have a horizon for that?

Softbear•1/26/24, 7:48 PM

The ability to export the database as a .sqlite was very useful, is there any plan to bring it back ?

Original message was deleted

Max (@rozenmd)•1/26/24, 9:25 PM

correction @WcaleNieWolny: D1 lives on the edge, but only one DB gets created near you. When replication ships, there'll be dbs around the world.

the D1 API (as documented here: https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database) lives centrally, and is slow to query as a result (and why we recommend querying D1 from a Worker), until we rewrite it to run from a Worker.

Will•1/27/24, 1:23 AM

Is there anyway to switch database dynamically? I am building a multi tenant app, and would like to just use one worker for each client

Pepper•1/27/24, 2:47 AM

suggestion: when adding data through the web ui, there should be a confirmation alert if you try to close the tab

angora•1/27/24, 5:49 AM

What is the best way to set up transfer of a set of tables from Google bigquery to D1 on daily basis

Bram•1/27/24, 11:32 AM

Hi all, have more of a general question, considering D1 is a new kind of database (basically replicated sqlite files at edge) and considering the very fast reads of SQLite, is there still a need to use traditional key value stores for perf / latency reasons?

kian•1/27/24, 11:59 AM

D1 isn’t replicated at edge currently

Samantas5855•1/27/24, 12:13 PM

Error inserting data for channel1: 500, {
"result": [],
"success": false,
"errors": [
{
"code": 7500,
"message": "logEvent(): Cumulative size of blobs exceeds 5120 bytes)."
}
],
"messages": []
}

Samantas5855•1/27/24, 12:13 PM

is that new?

Samantas5855•1/27/24, 12:14 PM

I never had this one before

Samantas5855•1/27/24, 12:30 PM

my script can't update my d1 db since 9 hours ago and my app is now broken because of this

Samantas5855•1/27/24, 12:31 PM

changelog mentions nothing and the only identical limit I could find is on analytics

SSamantas5855 Error inserting data for channel1: 500, { "result": [], "success": false, ...

Max (@rozenmd)•1/27/24, 2:22 PM

that looks like a bug, will investigate when I'm at a laptop

WcaleNieWolnyOP•1/27/24, 2:58 PM

Can some please explain to me how on earth did i get 4 billion writes to the db? I included some screenshots of the usage reported in different places. I do not get this at all. I get the reads but not the writes

WcaleNieWolnyOP•1/27/24, 3:00 PM

Yeah i get that

I rely heavily on the [d1 http api](https://developers.cloudflare.com/api/operations/cloudflare-d1-q

Similar Threads