I need to return a response from the worker asap and also run a write query to D1. Is it possible to

RRob M.I need to return a response from the worker asap and also run a write query to D...

S

Sam•11/27/23, 12:19 PM

https://developers.cloudflare.com/workers/runtime-apis/handlers/fetch/#contextwaituntil

RRob M.250ms from the API or worker?

J

johnswarbrick•11/27/23, 1:29 PM

250ms is the time taken for the D1 database to respond with results back to my worker. Sometimes dips to 210ms, but it's quite consistent. Really hoping having read-replicas speeds this up.

Jjohnswarbrick 250ms is the time taken for the D1 database to respond with results back to my w...

M

Max (@rozenmd)•11/27/23, 1:39 PM

Are you using indexes?

M

Max (@rozenmd)•11/27/23, 1:41 PM

because fetching a specific record in the northwind db takes 0.2ms: https://northwind.d1sql.com/dash

J

johnswarbrick•11/27/23, 1:50 PM

Yes, I think it's networking latency.

The Cloudflare Worker is being spun up in the UK, because that's the closest CF endpoint.

But then the SQL query is being sent to wherever the D1 core is currently.

So even a trivial 1 row select from a 1 row table takes 200 - 250ms to return.

I've not tested from APAC yet, I guess it will end up 500-750ms from there.

I'm really hoping read-replicas fixes that problem, although I guess writes will still be slow.

Jjohnswarbrick Yes, I think it's networking latency. The Cloudflare Worker is being spun up in...

M

Matt Silverlock•11/27/23, 1:55 PM

D1 will locate the leader close to where you created it, or if you provide a hint, close to where you request: https://developers.cloudflare.com/d1/learning/data-location/

Jjohnswarbrick Yes, I think it's networking latency. The Cloudflare Worker is being spun up in...

M

Matt Silverlock•11/27/23, 1:56 PM

You can see where your leader is located with

wrangler d1 info DB_NAME

wrangler d1 info DB_NAME

wrangler d1 info DB_NAME

wrangler d1 info DB_NAME

e.g.

``
➜ npx wrangler@latest d1 info db-enam
┌───────────────────┬──────────────────────────────────────┐
│ ENAM │ c89db32e-83f4-4e62-8cd7-7c8f97659029 │
├───────────────────┼──────────────────────────────────────┤
│ name │ db-enam │
├───────────────────┼──────────────────────────────────────┤
│ created_at │ 2023-06-12T16:52:03.071Z │
├───────────────────┼──────────────────────────────────────┤
│ version │ beta │
├───────────────────┼──────────────────────────────────────┤
│ num_tables │ 8 │
├───────────────────┼──────────────────────────────────────┤
│ running_in_region │ ENAM │
├───────────────────┼──────────────────────────────────────┤
│ database_size │ 1.83 GB │
├───────────────────┼──────────────────────────────────────┤
│ read_queries_24h │ 1,437 │
├───────────────────┼──────────────────────────────────────┤
│ write_queries_24h │ 1,551,235 │
└───────────────────┴──────────────────────────────────────┘
```

J

johnswarbrick•11/27/23, 2:01 PM

Thanks Matt, I'll take a look

SSantosh can someone provide an example on this topic: How do I insert and then append a ...

S

Santosh•11/27/23, 2:46 PM

Can someone give me an example on how to insert and append Json using json_insert? I see the example from the documentation with scalar but not an object. And i receive the following error :

Error: D1_TYPE_ERROR: Type 'object' not supported for value '[object Object]'
    at cloudflare-internal:d1-api:146:19

Error: D1_TYPE_ERROR: Type 'object' not supported for value '[object Object]'
    at cloudflare-internal:d1-api:146:19

Error: D1_TYPE_ERROR: Type 'object' not supported for value '[object Object]'
    at cloudflare-internal:d1-api:146:19

Error: D1_TYPE_ERROR: Type 'object' not supported for value '[object Object]'
    at cloudflare-internal:d1-api:146:19

SSantosh Can someone give me an example on how to insert and append Json using json_inse...

M

Matt Silverlock•11/27/23, 4:16 PM

Can you show what you have tried first?

S

Santosh•11/27/23, 5:49 PM

      const h = {id: 1, attr1: "abcdef} const t = JSON.stringify(h)
      //   console.log("JSON being inserted =>>>>", h)
        const result = await env.SMS_DEV.prepare(
          "UPDATE TestTable SET history = json_insert(history, '$.history[#]', '?1') WHERE id = '?2' ",
        )
          .bind(h, status.id)
          .run()

      const h = {id: 1, attr1: "abcdef} const t = JSON.stringify(h)
      //   console.log("JSON being inserted =>>>>", h)
        const result = await env.SMS_DEV.prepare(
          "UPDATE TestTable SET history = json_insert(history, '$.history[#]', '?1') WHERE id = '?2' ",
        )
          .bind(h, status.id)
          .run()

      const h = {id: 1, attr1: "abcdef} const t = JSON.stringify(h)
      //   console.log("JSON being inserted =>>>>", h)
        const result = await env.SMS_DEV.prepare(
          "UPDATE TestTable SET history = json_insert(history, '$.history[#]', '?1') WHERE id = '?2' ",
        )
          .bind(h, status.id)
          .run()

      const h = {id: 1, attr1: "abcdef} const t = JSON.stringify(h)
      //   console.log("JSON being inserted =>>>>", h)
        const result = await env.SMS_DEV.prepare(
          "UPDATE TestTable SET history = json_insert(history, '$.history[#]', '?1') WHERE id = '?2' ",
        )
          .bind(h, status.id)
          .run()

S

Santosh•11/27/23, 7:21 PM

Can someone point to Full Test search capabilities of D1? I couldn't locate any articles or discussions related to this

W

WoozyAnuki•11/27/23, 7:56 PM

what's the format for running SQL queries on D1 on PHP deployed as a project? I'm an absolute noob with web dev but using cloudflare to host a student project as well as my portfolio but for the project I'm requried to use SQL and saw D1 as an option so was wondering if it was possible to use d1 as I would use a normal SQL database (as in like XAMPP/MAMP)

WWoozyAnuki what's the format for running SQL queries on D1 on PHP deployed as a project? I'...

H

Hello, I’m Allie!•11/27/23, 8:04 PM

You can’t deploy PHP on Cloudflare

H

Hello, I’m Allie!•11/27/23, 8:04 PM

Or well, I think there is an adapter for it somewhere, but not sure it supports anything relatively modern

H

Hello, I’m Allie!•11/27/23, 8:05 PM

Or the PHP runtime someone wrote.

W

WoozyAnuki•11/27/23, 8:07 PM

Woof, thanks lol. Will still use it for my portfolio and just use R2 most likely since there's no extra mess lol. Tysm :)

SSantosh Can someone point to Full Test search capabilities of D1? I couldn't locate any ...

K

kian•11/27/23, 8:32 PM

It's SQLite's FTS5

SSantosh Can someone point to Full Test search capabilities of D1? I couldn't locate any ...

R

Rob M.OP•11/27/23, 10:16 PM

Look at this one:
https://www.sqlite.org/fts5.html

S

scotto•11/27/23, 10:43 PM

hi - any github example for a worker project with d1? need to build for mid-size project and a good example on how to structure the code would be great to start

RRob M.Look at this one: https://www.sqlite.org/fts5.html

S

Santosh•11/28/23, 2:03 AM

How do we use this on D1? An example would be perfect, not many of use SQLLite at work.

V

Village (Josef)•11/28/23, 4:59 AM

Two questions

Are all the limits except the DB size limit available to be potentially raised?
Is it feasible to make a DB per customer with D1? I.e. is there some way to dynamically set up bindings and such without having to manually put them all inside wrangler.toml and without incurring insane startup overhead or having to route through the API? (That was a lot of or’s sorry)

K

kian•11/28/23, 5:31 AM

Are all the limits except the DB size limit available to be potentially raised?

You'd need to be specific about the individual limit in question, but generally they aren't raised on a per-user basis and instead are global limits.

is there some way to dynamically set up bindings

Not currently.

R

Ravi kiran•11/28/23, 5:36 AM

guys can anyone tell me how i can use two d1 databases in a single worker?
i tried this in the toml file
name = "weathered-queen-4070"
main = "src/index.ts"
compatibility_date = "2023-10-30"

[[d1_databases]]
binding = "DB" # available in your Worker on env.DB
database_name = "dogpark"
database_id = "xxxxxxxxxxxx"

binding = "DB2" # i.e. available in your Worker on env.DB
database_name = "adminportalDB"
database_id = "xxxxxxxxxxx"

but then i get the error that Can't redefine existing key

/Users/apple/Documents/weathered-queen-4070/wrangler.toml:10:14:
10 │ binding = "DB2"

K

kian•11/28/23, 5:37 AM

name = "weathered-queen-4070"
main = "src/index.ts"
compatibility_date = "2023-10-30"

[[d1_databases]]
binding = "DB" # available in your Worker on env.DB
database_name = "dogpark"
database_id = "xxxxxxxxxxxx"

[[d1_databases]]
binding = "DB2" # i.e. available in your Worker on env.DB
database_name = "adminportalDB"
database_id = "xxxxxxxxxxx"

name = "weathered-queen-4070"
main = "src/index.ts"
compatibility_date = "2023-10-30"

[[d1_databases]]
binding = "DB" # available in your Worker on env.DB
database_name = "dogpark"
database_id = "xxxxxxxxxxxx"

[[d1_databases]]
binding = "DB2" # i.e. available in your Worker on env.DB
database_name = "adminportalDB"
database_id = "xxxxxxxxxxx"

name = "weathered-queen-4070"
main = "src/index.ts"
compatibility_date = "2023-10-30"

[[d1_databases]]
binding = "DB" # available in your Worker on env.DB
database_name = "dogpark"
database_id = "xxxxxxxxxxxx"

[[d1_databases]]
binding = "DB2" # i.e. available in your Worker on env.DB
database_name = "adminportalDB"
database_id = "xxxxxxxxxxx"

name = "weathered-queen-4070"
main = "src/index.ts"
compatibility_date = "2023-10-30"

[[d1_databases]]
binding = "DB" # available in your Worker on env.DB
database_name = "dogpark"
database_id = "xxxxxxxxxxxx"

[[d1_databases]]
binding = "DB2" # i.e. available in your Worker on env.DB
database_name = "adminportalDB"
database_id = "xxxxxxxxxxx"

R

Ravi kiran•11/28/23, 5:38 AM

oh thank you.

R

rodrigo•11/28/23, 8:55 AM

I understand when read replicas are implemented there would be one master where all writes take place, right?
Is there any plans to have multiple write masters, based on table partition (for instance)?
Imagine a User table, where users write their own data. Would it be possible to partition this table based on location?
I mean, to create a partition of this table for each location (america, europe, asia) that would become master for writes, and this way a user would write as master to the partition in its location (and then read replica would replicate it to other locations)

V

VI•11/28/23, 9:48 AM

Since d1 uses SQLite internally, does it run into the same limitations as SQLite, namely handling large number of independent writes/sec. From what I read SQLite is not capable of handling more than 1000 write reqs/sec.
https://www.sqlite.org/whentouse.html

Thanks for any help!

Rrodrigo I understand when read replicas are implemented there would be one master where ...

M

Max (@rozenmd)•11/28/23, 8:12 PM

So far the plan is just for a single writer

VVI Since d1 uses SQLite internally, does it run into the same limitations as SQLite...

V

Vy•11/28/23, 8:32 PM

Currently for open beta, ~1K QPS depending on result set size. Our eventual goal is to get 3-10K QPS for reads, 1-2K QPS for writes. Let us know what requirements your workload would need?

Kkian > Are all the limits except the DB size limit available to be potentially raised...

V

Village (Josef)•11/29/23, 3:06 AM

Sounds good, thank you! And I meant storage per account and number of DBs specifically

J

janat08•11/29/23, 8:51 AM

Can you promise at this point that the D1 size limitations will begone

J

janat08•11/29/23, 8:52 AM

You recommend against building on D1 for production, but surely not because you still have forseeable hard limits on db size?

Jjanat08 You recommend against building on D1 for production, but surely not because you ...

H

Hello, I’m Allie!•11/29/23, 8:54 AM

I don't think it is ever going to be unlimited, but it is definitely going to be higher than it is now

J

janat08•11/29/23, 9:01 AM

Do you get to shard it or something once at the ceiling. You mean I won't be able to have a petabyte db on cf, but my question is if at some point I'd have to close registration at my app because of the storage limit?

Jjanat08 Do you get to shard it or something once at the ceiling. You mean I won't be abl...

H

Hello, I’m Allie!•11/29/23, 9:04 AM

I think it is also a case of understanding what platforms are built for. While it would be nice theoretically if D1 could scale infinitely, in practice, if you do get to Petabyte scale, you should probably look at a DB specfically built for that. Until then though, D1 is a good choice

J

janat08•11/29/23, 9:22 AM

Is there going to be some library for sharding, where you can have multiple d1 dbs, and get to treat them one, so as to bypass the size limitation at the expense of extra reads

M

Max (@rozenmd)•11/29/23, 9:28 AM

we have folks building products that have 1 db per customer, where each db has the same schema, just stores different data
though migrations + dynamically binding to new dbs are a pain point

MMax (@rozenmd)we have folks building products that have 1 db per customer, where each db has t...

H

Hello, I’m Allie!•11/29/23, 9:33 AM

Where's my SQLite DOs?

J

janat08•11/29/23, 9:34 AM

I'm not interested in 1db per customer thing, as I'd like them to be able to interact with each other

J

janat08•11/29/23, 9:34 AM

is there some kind of workaround, or limitations improvement to make d1 fit for general usage?

Jjanat08 is there some kind of workaround, or limitations improvement to make d1 fit for ...

M

Max (@rozenmd)•11/29/23, 9:40 AM

It doesn't sound like D1 fits your use case if you're worried about petabyte scale. Hyperdrive is probably a better fit: https://developers.cloudflare.com/hyperdrive/

J

janat08•11/29/23, 10:38 AM

I have no reason to worry about petabyte scyle, but surely anyone is hitting the 2gb limit

K

kian•11/29/23, 11:04 AM

This is a beta-only limitation. The D1 team will increase the maximum storage per-database, storage per-account and number of databases for Paid plans during the course of D1’s public beta.

K

kian•11/29/23, 11:05 AM

They're beta limitations - there is no forecasted hard limit, but there will naturally be one even in GA.

J

janat08•11/29/23, 12:13 PM

612345 wtyu wq\vcb

M

mushang98•11/29/23, 12:30 PM

I'm sorry for asking. I'm a newbie around cloudflare workers and pages. I've been trying to use next.js with D1 with no luck. Is there any template or example code and configs for this ?

I need to return a response from the worker asap and also run a write query to D1. Is it possible to

Similar Threads

Similar Threads

Similar Threads