https://vxtwitter.com/elithrar/status/1765452226460340591?s=20

MMatt Silverlock https://vxtwitter.com/elithrar/status/1765452226460340591?s=20

A

Alsquiaga•3/22/24, 3:02 PM

Thanks a lot! I am very excited to start using these features for my projects!

M

Marcel Overdijk•3/22/24, 6:43 PM

Hi,

Recently I started experimenting with Cloudflare to host a website (static) and a dynamic REST API via workers.
Behind the REST API there is a database with approx. 200K rows which I need to periodically import into D1

So last week I did the initial import using Wrangler and it went really smoothly

A few minutes ago I tried to re-import and it failed with:

✘ [ERROR] Received a malformed response from the API

  <!DOCTYPE html>
  <!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
  <!--[if I... (length = 7300)

  POST
  /accounts/xxxxxxxxxxxxxxxxxxxxxxx/d1/database/xxxxxxxxxxxxxxxxxxxxxxx/query
  -> 520

  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

✘ [ERROR] Received a malformed response from the API

  <!DOCTYPE html>
  <!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
  <!--[if I... (length = 7300)

  POST
  /accounts/xxxxxxxxxxxxxxxxxxxxxxx/d1/database/xxxxxxxxxxxxxxxxxxxxxxx/query
  -> 520

  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

✘ [ERROR] Received a malformed response from the API

  <!DOCTYPE html>
  <!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
  <!--[if I... (length = 7300)

  POST
  /accounts/xxxxxxxxxxxxxxxxxxxxxxx/d1/database/xxxxxxxxxxxxxxxxxxxxxxx/query
  -> 520

  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

✘ [ERROR] Received a malformed response from the API

  <!DOCTYPE html>
  <!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
  <!--[if I... (length = 7300)

  POST
  /accounts/xxxxxxxxxxxxxxxxxxxxxxx/d1/database/xxxxxxxxxxxxxxxxxxxxxxx/query
  -> 520

  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

My import script first executes some

DROP TABLE IF EXISTS ..

DROP TABLE IF EXISTS ..

statements, then creates the tables, and executes the inserts.
Although it failed, all tables were deleted, created and also some data was inserted.

It seems to have failed after approx. 25K rows out of 175K.

Are there any log files or something else I can check to see what happened?

Note I can import this file without any problem in a local sqlite database.

M

Marcel Overdijk•3/22/24, 6:45 PM

Another thing I now noticed is this in my D1 account page:

Rows read
1.25M

Rows written
375.35k

Rows read
1.25M

Rows written
375.35k

Before starting the import it said

and

.
So I can not really explain these numbers as now only ~25K rows were inserted...

X

Xazu•3/22/24, 7:06 PM

Hi! I have a question. So how much data i can store on single db?

Screenshot_2024-03-22-20-02-29-080_com.android.chrome.jpg

X

Xazu•3/22/24, 7:08 PM

Is the limits page up-to-date?

C

Chaika•3/22/24, 7:08 PM

The first screenshot is showing pricing. On Paid, you get 5 GB included. The max DB size, as you see in the second screenshot, is different. 2 GB for paid per database.

XXazu Is the limits page up-to-date?

C

Chaika•3/22/24, 7:09 PM

yes, although they're working on bringing it up to 10 GB d1-database

C

Chaika•3/22/24, 7:09 PM

D1, as of right now, wants you to split your data among many databases rather then have one big one, thus the reasons of the limits being like that

X

Xazu•3/22/24, 7:10 PM

Ok. Thank you for the answer

S

sirhype•3/22/24, 9:35 PM

Can't figure out what I'm missing here. I've got two tables made. I'm using Drizzle ORM, and I'm trying to run some queries. But I get

D1_ERROR: no such table: inventory

D1_ERROR: no such table: inventory

.

Any ideas where I might be going wrong? The statement it's trying to run is:

{
  sql: 'delete from "inventory" where ("inventory"."user_id" = ? and "inventory"."skin_id" not in (?, ?, ?, ?))',
  params: [
    '38891062',
    '9fa5664e-45fa-2a52-68e4-db97d85c8a91',
    '7e323e82-44b6-1711-2028-10af32c632d5',
    '19ba907b-4a76-347f-ea24-38990c9ff755',
    '7b71fc11-4fa2-c84c-31ec-5eb5c3e3fff3'
  ]
}

{
  sql: 'delete from "inventory" where ("inventory"."user_id" = ? and "inventory"."skin_id" not in (?, ?, ?, ?))',
  params: [
    '38891062',
    '9fa5664e-45fa-2a52-68e4-db97d85c8a91',
    '7e323e82-44b6-1711-2028-10af32c632d5',
    '19ba907b-4a76-347f-ea24-38990c9ff755',
    '7b71fc11-4fa2-c84c-31ec-5eb5c3e3fff3'
  ]
}

{
  sql: 'delete from "inventory" where ("inventory"."user_id" = ? and "inventory"."skin_id" not in (?, ?, ?, ?))',
  params: [
    '38891062',
    '9fa5664e-45fa-2a52-68e4-db97d85c8a91',
    '7e323e82-44b6-1711-2028-10af32c632d5',
    '19ba907b-4a76-347f-ea24-38990c9ff755',
    '7b71fc11-4fa2-c84c-31ec-5eb5c3e3fff3'
  ]
}

{
  sql: 'delete from "inventory" where ("inventory"."user_id" = ? and "inventory"."skin_id" not in (?, ?, ?, ?))',
  params: [
    '38891062',
    '9fa5664e-45fa-2a52-68e4-db97d85c8a91',
    '7e323e82-44b6-1711-2028-10af32c632d5',
    '19ba907b-4a76-347f-ea24-38990c9ff755',
    '7b71fc11-4fa2-c84c-31ec-5eb5c3e3fff3'
  ]
}

Creating the DB instance with:

  if (!env.DB) {
    throw new Error("Database connection not found");
  }
  const db = drizzle(env.DB, {
    schema,
    logger: true,
  });

  if (!env.DB) {
    throw new Error("Database connection not found");
  }
  const db = drizzle(env.DB, {
    schema,
    logger: true,
  });

  if (!env.DB) {
    throw new Error("Database connection not found");
  }
  const db = drizzle(env.DB, {
    schema,
    logger: true,
  });

  if (!env.DB) {
    throw new Error("Database connection not found");
  }
  const db = drizzle(env.DB, {
    schema,
    logger: true,
  });

Ssirhype Can't figure out what I'm missing here. I've got two tables made. I'm using Driz...

M

Max (@rozenmd)•3/22/24, 10:15 PM

bump wrangler to latest, explicitly use either

--local

--local

or

--remote

--remote

J

Jacob Wright•3/22/24, 10:22 PM

Developer week is mid-May, right? Any hints for what we might get with D1? GA, multi-region replication, DurableObject-local Sqlite?

MMax (@rozenmd)bump wrangler to latest, explicitly use either `--local` or `--remote`

S

sirhype•3/22/24, 10:43 PM

ahhh the

--remote

--remote

was what I was missing. Thank you!

T

Tony Leung•3/23/24, 10:47 AM

hi guys, is there any cache solution for D1 select query? With MySQL I can cache query by using ProxySQL.

P

PohKit•3/23/24, 5:01 PM

Hi guys,

Was exploring options between D1, PlanetScale & ManagedDB by Digital Ocean.

Just wanted to check has anyone tried with D1 x PHP/Laravel with this repo:
https://github.com/renoki-co/l1

I've created a Laravel project, written some basic API.

But seems like the Latency is high eg: 1,500ms ~ 2,000ms

Just wanted to confirm that, is D1 Laravel/PHP friendly?

GitHub

GitHub - renoki-co/l1: Extend your PHP/Laravel application with Clo...

Extend your PHP/Laravel application with Cloudflare bindings. - renoki-co/l1

TTony Leung hi guys, is there any cache solution for D1 select query? With MySQL I can cache...

S

squareclamp•3/23/24, 11:34 PM

Not sure if there is anything out of the box, but you can look at the cache api - https://developers.cloudflare.com/workers/runtime-apis/cache/

Cloudflare Docs

Cache · Cloudflare Workers docs

Control reading and writing from the Cloudflare global network cache.

X

x03•3/24/24, 4:12 AM

How do I apply migrations on my production database? To apply local migrations, I just run:

bun wrangler d1 migrations apply db_preview --local

bun wrangler d1 migrations apply db_preview --local

This is a SvelteKit pages app.

X

x03•3/24/24, 4:13 AM

And my wrangler.toml if needed:

[[d1_databases]]
binding = "DB"
database_name = "db_preview"
database_id = "db_preview"
migrations_dir = "./drizzle"

[[d1_databases]]
binding = "DB"
database_name = "db_preview"
database_id = "db_preview"
migrations_dir = "./drizzle"

[[d1_databases]]
binding = "DB"
database_name = "db_preview"
database_id = "db_preview"
migrations_dir = "./drizzle"

[[d1_databases]]
binding = "DB"
database_name = "db_preview"
database_id = "db_preview"
migrations_dir = "./drizzle"

Xx03 How do I apply migrations on my production database? To apply local migrations, ...

P

predaytor•3/24/24, 9:49 AM

there is

--remote

--remote

flag

M

melodyclue•3/24/24, 2:25 PM

Hi, Is this compatible with remix+vite?　https://developers.cloudflare.com/d1/examples/d1-and-remix/

Cloudflare Docs

Query D1 from Remix · Cloudflare D1 docs

Remix is a full-stack web framework that operates on both client and server. You can query your D1 database(s) from Remix using Remix’s data loading …

M

mw10013•3/24/24, 3:38 PM

Yes, that works with remix+vite. I found that it’s not necessary to specify the bindings in the command line when running locally. Just let ‘remix vite:dev’ pick them up from wrangler.toml and include the local bindings in the wrangler.toml. I put together the cloudflare example for ‘remix-auth-totp’ that uses drizzle with d1 that has the local bindings in wrangler.toml. https://github.com/mw10013/remix-auth-totp-cloudflare-example

GitHub

GitHub - mw10013/remix-auth-totp-cloudflare-example

Contribute to mw10013/remix-auth-totp-cloudflare-example development by creating an account on GitHub.

F

F0rce•3/24/24, 6:16 PM

I thought this was fixxed some time ago :MonkaHmm: https://github.com/cloudflare/workers-sdk/issues/3160#issuecomment-2010837504

GitHub

🐛 BUG: when trying to left join 2 tables that have same name column...

Which Cloudflare product(s) does this pertain to? D1 What version of Wrangler are you using? 2.19.0 What operating system are you using? macOS Describe the Bug As per issue #555 we can see, that d1...

FF0rce I thought this was fixxed some time ago :MonkaHmm: https://github.com/cloudflare...

M

Matt SilverlockOP•3/24/24, 9:44 PM

Just a bug in local mode (unfortunately). raw() was fixed back in Feb https://developers.cloudflare.com/d1/platform/changelog/#api-changes-to-raw-all-and-run

Cloudflare Docs

Changelog · Cloudflare D1 docs

As of wrangler@3.33.0, wrangler d1 execute and wrangler d1 migrations apply now default to using a local database, to match the default behavior of …

MMatt Silverlock Just a bug in local mode (unfortunately). raw() was fixed back in Feb https://de...

F

F0rce•3/24/24, 9:45 PM

wait so local development and remote development (in this scenario) return different results ?

F

F0rce•3/24/24, 9:45 PM

i knew that it was fixxed, but I would not deploy to prod without proper testing locally.

B

bun•3/25/24, 2:17 AM

does

count(*)

count(*)

consume reads on how many items there actually are counted ?

A

Aim Salam•3/25/24, 7:10 AM

#d1-beta is it possible cloudflare D1 will have unlimited size per database?

AAim Salam #d1-beta is it possible cloudflare D1 will have unlimited size per database?

D

DaniFoldi•3/25/24, 8:28 AM

unlikely in my opinion to have unlimited sizes anytime soon

AAim Salam #d1-beta is it possible cloudflare D1 will have unlimited size per database?

M

Max (@rozenmd)•3/25/24, 8:55 AM

D1 is built to work to the strengths of Cloudflare's network, so we can't offer extremely large databases while also loading them quickly. It's significantly easier for us to offer millions of DBs with a size limit instead.

The limit is currently 2GB and we're working towards making it 10GB - but D1 will require a change in how you think about your project's data if you're looking to build a big platform. Instead of storing all of your data in a single DB, you would store each customer's data in their own DB (and we're building tooling to make it easier to dynamically access multiple DBs in a single Worker without redeploying)

MMax (@rozenmd)D1 is built to work to the strengths of Cloudflare's network, so we can't offer ...

A

Aim Salam•3/25/24, 8:58 AM

Thank you for the response. I'm very excited what D1 can do in the future. I'm looking forward to it

M

Max (@rozenmd)•3/25/24, 8:59 AM

also if you want to stick to the old model (one single big db), we have Hyperdrive for that: https://developers.cloudflare.com/hyperdrive/

R

Revan•3/25/24, 9:01 AM

hi, does cf d1 support/implement the

generate_series

generate_series

function of sqlite?

RRevan hi, does cf d1 support/implement the `generate_series` function of sqlite?

M

Max (@rozenmd)•3/25/24, 9:06 AM

doesn't look like we've enabled that extension yet, if you file a feature request for it here, that'd be awesome: https://github.com/cloudflare/workerd/issues

R

Revan•3/25/24, 9:11 AM

sure, will do

R

Revan•3/25/24, 9:21 AM

I made the issue here: https://github.com/cloudflare/workerd/issues/1888

wasnt really sure what to write

RRevan I made the issue here: https://github.com/cloudflare/workerd/issues/1888 wasnt ...

M

Max (@rozenmd)•3/25/24, 9:53 AM

perfect, thank you!

S

sabin•3/25/24, 1:42 PM

how does d1 compare to turso?

S

sabin•3/25/24, 1:42 PM

couldnt find anything online abt it

AAaronO Could you elaborate a little on the eager load proportional to DB size ? As an ...

M

Max (@rozenmd)•3/25/24, 5:55 PM

Probably a better question for @kenton (when they're around)

MMax (@rozenmd)D1 is built to work to the strengths of Cloudflare's network, so we can't offer ...

I

itsmatteomanf•3/25/24, 5:58 PM

The biggest limit here (but I know you are working on it, so hopefully soon it gets fixed) is that you can't dynamically connect to a database.
If I have millions of users, how can I configure a database per user (if I can work around the limits that cross-user queries are impossible, at the moment) and reach it via Workers?

AAaronO Could you elaborate a little on the eager load proportional to DB size ? As an ...

K

kenton•3/25/24, 6:10 PM

The way you are imagining it works is the way D1 v1 and v2 worked... which turned out to be way too slow. In v3 we pull the full database down to local disk upfront.

AAaronO Did v1 and v2 use the local disk as cache too ? In the limit, eagerly pulling th...

K

kenton•3/25/24, 6:19 PM

I believe pages were actually cached in RAM (which is why the database size limit was 100MB). I wasn't closely involved with v1 or v2 though so can't comment in that much detail.

K

kenton•3/25/24, 6:20 PM

v3 is based on a new storage engine which I described a bit here: https://twitter.com/KentonVarda/status/1659551757796515846

Kenton Varda (@KentonVarda) on X

So I've spent the last ~three months writing a new distributed storage engine based on SQLite and it has been FUN!

https://t.co/NVorwS46x8

Twitter

•

5/19/23, 1:29 PM

K

kenton•3/25/24, 6:20 PM

(still haven't actually published the alluded blog post, sorry)

D

dave•3/25/24, 7:21 PM

is there a way to skip to the end of a table in the Cloudflare Dashboard?

https://vxtwitter.com/elithrar/status/1765452226460340591?s=20

Similar Threads