Cloudflare Developers•3y ago

If you need performance, write a Worker then talk to it over HTTP from python

MMax (@rozenmd)If you need performance, write a Worker then talk to it over HTTP from python

Matt Silverlock•10/10/23, 6:03 PM

https://github.com/elithrar/http-api-d1-example

GitHub

GitHub - elithrar/http-api-d1-example: An example HTTP API for Clou...

An example HTTP API for Cloudflare's D1 database: https://developers.cloudflare.com/d1/

GitHub - elithrar/http-api-d1-example: An example HTTP API for Cloudflare's D1 database: ht...

andyt•10/10/23, 6:55 PM

Hi, I am running wrangler pages dev --compatibility-date=2023-01-01 ./pages --kv=KV --d1=DBwrangler pages dev --compatibility-date=2023-01-01 ./pages --kv=KV --d1=DB and it cannot find my table, I have assumed the d1 binding is to my remote database? is this not the case?

andyt•10/10/23, 6:58 PM

ah ok, how do you create a local d1 database and script it?

Cretezy•10/10/23, 9:05 PM

Why is the D1 HTTP API lower performance than the API in Workers? (excluding that it's not in the same datacenter)

Original message was deleted

Cretezy•10/10/23, 9:10 PM

That's the "(excluding that it's not in the same datacenter)" part, but it seems like the API is quite slow, even for remote

Cretezy•10/10/23, 9:12 PM

I'm seeing 500-700ms for simple queries

Max (@rozenmd)OP•10/10/23, 9:12 PM

The API isn't designed for Workers-style access patterns, it's an API

Max (@rozenmd)OP•10/10/23, 9:13 PM

whereas a Worker will be within 50ms of 95% of the internet-connected world

Cretezy•10/10/23, 9:15 PM

But why isn't it designed for this? Multiple queries in a Worker is <100ms, a single of those queries outside is >500ms. They are designed for Workers, but the API being 5x slower seems like a missed opportunity

Cretezy•10/10/23, 9:15 PM

The alternative is making a query worker such as http-api-d1-examplehttp-api-d1-example above, but it feels like the API should be able to do that

Original message was deleted

itsmatteomanf•10/10/23, 9:27 PM

This, and also the core datacenter might be distant from the D1 location, which adds to the roundtrip time.

itsmatteomanf•10/10/23, 9:27 PM

You are generally close to the database you create, so it's fast you -> worker -> db.

itsmatteomanf•10/10/23, 9:28 PM

If you use the API you do you -> api (possibly far away) -> worker (it's an assumption, but I bet it's a worker or similar) -> db (close to you).

itsmatteomanf•10/10/23, 9:29 PM

This makes queries at least 4x the time to go from you to the API, plus inefficiencies as connections need to be established.

Iitsmatteomanf This, and also the core datacenter might be distant from the D1 location, which ...

Max (@rozenmd)OP•10/10/23, 9:42 PM

this, we built an API rapidly with what we already had, we might evolve it in the future but that's what's there today

Paul•10/10/23, 9:46 PM

Any guis work with D1 for viewing live data?

Paul•10/10/23, 9:47 PM

(It seems I can use dbeaver for viewing the local SQLite file)

PPaul Any guis work with D1 for viewing live data?

Cyb3r-Jak3•10/10/23, 9:57 PM

There is a console in the dashboard

CCyb3r-Jak3 There is a console in the dashboard

Paul•10/10/23, 10:01 PM

Are you able to run arbitrary queries from the dashboard?

PPaul Are you able to run arbitrary queries from the dashboard?

Cyb3r-Jak3•10/10/23, 10:01 PM

Yeah. Go to your D1 database in the dashboard and there will a tab of ConsoleConsole

Paul•10/10/23, 10:19 PM

Ok great. Is there a plan to support more traditional non web based clients

PPaul Ok great. Is there a plan to support more traditional non web based clients

Cyb3r-Jak3•10/10/23, 10:24 PM

Personally, I wouldn't expect any official support given that D1 would only be queryable over HTTP but people from the community might add support to the various projects

CCyb3r-Jak3 Personally, I wouldn't expect any official support given that D1 would only be q...

Hello, I’m Allie!•10/10/23, 10:43 PM

Is HTTP even officially supported? I know it is wrangler, but I forget if it is documented

kian•10/10/23, 10:44 PM

Yeah

kian•10/10/23, 10:44 PM

https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database

Cyb3r-Jak3•10/10/23, 10:54 PM

^ Came with upgrade to beta

ChrisBrocklesby•10/10/23, 10:55 PM

Hi, I am having major issues uploading a large amount of data from a .sql dump to D1 "136,000 rows" any tips / help you guys can offer ?

CChrisBrocklesby Hi, I am having major issues uploading a large amount of data from a .sql dump t...

Cyb3r-Jak3•10/10/23, 11:00 PM

You could try splitting the sql dump file into multiple files and do each one separately

CCyb3r-Jak3 You could try splitting the sql dump file into multiple files and do each one se...

ChrisBrocklesby•10/10/23, 11:20 PM

Hi, I have tried splitting the 650mb .sql file down into 2mb files with no luck sadly. I get the following error via Wrangler...

Paul•10/11/23, 12:03 AM

anyone using SvelteKit managed to get

npm run dev

npm run dev

working after adding D1? I'm assuming we would need to tell it to run the local db

Fabian•10/11/23, 1:51 AM

How to select the most recent day's data from the database?
The database content is as follows:
id ip UA time
41 139.162.86.143 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36 2023/10/7 09:40:22

Can this be done?
SELECT * FROM m_table
WHERE time >= DATE_SUB(NOW(), INTERVAL 1 DAY);

Fabian•10/11/23, 1:52 AM

Original message was deleted

MrXO•10/11/23, 2:55 AM

use API

MMax (@rozenmd)If you need performance, write a Worker then talk to it over HTTP from python

MrXO•10/11/23, 2:56 AM

so we will use python -> call HTTP request to worker -> worker handle d1 db logic?

MMrXO use API

Fabian•10/11/23, 2:57 AM

大佬帮看看吧

MrXO•10/11/23, 2:58 AM

like this

curl --request POST \
--url https://api.cloudflare.com/client/v4/accounts/account-identifier/d1/database/database-identifier/query \
--header 'Authorization: Bearer undefined' \
--header 'Content-Type: application/json' \
--data '{
"params": [
"firstParam",
"secondParam"
],
"sql": "SELECT * FROM m_table
WHERE time >= DATE_SUB(NOW(), INTERVAL 1 DAY);"
}'

MrXO•10/11/23, 2:58 AM

change your --header 'Authorization: Bearer undefined' \

DDaniFoldi I recommend drizzle, we recently started migrating all projects to it and it wor...

Paolo•10/11/23, 4:50 AM

I had a look and it does not seem to have models? It seems only to create data rows rather than instantiate classes.

soup•10/11/23, 7:52 AM

Hello!

Does anybody know if there's a way to natively insert a new row and get the auto incremented id from the insertion?

Ssoup Hello! Does anybody know if there's a way to natively insert a new row and get ...

Max (@rozenmd)OP•10/11/23, 7:59 AM

add

RETURNING *

RETURNING *

to the end of the query

Max (@rozenmd)OP•10/11/23, 8:00 AM

RETURNING id

RETURNING id

just for the id column

andyt•10/11/23, 8:55 AM

I have setup my local D1 using wrangler and --local, I have inserted a row and can retrieve the row using wrangler execute, when I run my function I get

Error: D1_ERROR: no such table: company

Error: D1_ERROR: no such table: company

, I have my D! config in my toml file and when I run my pages I pass

--d1=DB

--d1=DB

which matches the binding name in my toml, what have I done wrong ?

MrXO•10/11/23, 9:02 AM

took 0.2s to insert a row via API, so slow

MrXO•10/11/23, 9:02 AM

maybe I need use worker

MMrXO maybe I need use worker

itsmatteomanf•10/11/23, 9:02 AM

This, yes.

MMrXO took 0.2s to insert a row via API, so slow

itsmatteomanf•10/11/23, 9:03 AM

The insert didn't take 200ms, the full API request did have a 200ms roundtrip+processing time.

MrXO•10/11/23, 9:03 AM

yes

itsmatteomanf•10/11/23, 9:03 AM

The time to insert I presume is returned in the metadata along with the query results (I assume, never used that API myself)

If you need performance, write a Worker then talk to it over HTTP from python

Similar Threads

Similar Threads

Similar Threads