a single DO instance can do around 1k requests per second, for complicated queries i've seen it drop

V

vakno•3/22/24, 10:04 AM

hello i need some help with indexes. Pretty new to this so sorry if this seems stupid.

i created table

  CREATE TABLE IF NOT EXISTS ProductData (
      id INTEGER PRIMARY KEY AUTOINCREMENT,
      url TEXT,
      price NUMBER,
      description TEXT
  )

  CREATE TABLE IF NOT EXISTS ProductData (
      id INTEGER PRIMARY KEY AUTOINCREMENT,
      url TEXT,
      price NUMBER,
      description TEXT
  )

  CREATE TABLE IF NOT EXISTS ProductData (
      id INTEGER PRIMARY KEY AUTOINCREMENT,
      url TEXT,
      price NUMBER,
      description TEXT
  )

  CREATE TABLE IF NOT EXISTS ProductData (
      id INTEGER PRIMARY KEY AUTOINCREMENT,
      url TEXT,
      price NUMBER,
      description TEXT
  )

then i created index

  'CREATE INDEX IF NOT EXISTS idx_url ON ProductData (price, description)'

  'CREATE INDEX IF NOT EXISTS idx_url ON ProductData (price, description)'

  'CREATE INDEX IF NOT EXISTS idx_url ON ProductData (price, description)'

  'CREATE INDEX IF NOT EXISTS idx_url ON ProductData (price, description)'

when i query table i get same numbers of row read as without index.

SELECT * FROM ProductData

SELECT * FROM ProductData

SELECT * FROM ProductData

SELECT * FROM ProductData

What am i doing wrong?

Vvakno hello i need some help with indexes. Pretty new to this so sorry if this seems s...

S

Sam•3/22/24, 10:27 AM

Because you’re not using a filter in the query. You’re basically requesting all rows with that query

SSam Because you’re not using a filter in the query. You’re basically requesting all ...

V

vakno•3/22/24, 10:38 AM

oh okay. Are there any other ways i can reduce rows read everytime route is triggered? Any caching systems?

MMax (@rozenmd)a single DO instance can do around 1k requests per second, for complicated queri...

S

Sam•3/22/24, 10:38 AM

Max mentions here that you could use the cache api @joker

SSam Max mentions here that you could use the cache api @joker

V

vakno•3/22/24, 10:40 AM

thanks!

A

a_sasin•3/22/24, 11:00 AM

How would I seed a local database every time it’s created? A migration would end up being applied to the prod db

Original message was deleted

S

Sam•3/22/24, 11:03 AM

https://developers.cloudflare.com/workers/runtime-apis/cache/

Cloudflare Docs

Cache · Cloudflare Workers docs

Control reading and writing from the Cloudflare global network cache.

Vvakno hello i need some help with indexes. Pretty new to this so sorry if this seems s...

M

Max (@rozenmd)OP•3/22/24, 11:27 AM

an index is only useful if you're looking for a specific record/types of records.

a good rule of thumb is, if you're using a WHERE clause on anything other than the PRIMARY KEY on a query that runs often, it might be worth indexing it

if you had a column like

user_id

user_id

user_id

user_id on

ProductData

ProductData

ProductData

ProductData, you could add an index for that column, and reduce your rows read when querying

ProductData

ProductData

ProductData

ProductData via

SELECT * FROM ProductData WHERE user_id = ?

SELECT * FROM ProductData WHERE user_id = ?

SELECT * FROM ProductData WHERE user_id = ?

SELECT * FROM ProductData WHERE user_id = ?

Aa_sasin How would I seed a local database every time it’s created? A migration would end...

M

Max (@rozenmd)OP•3/22/24, 11:29 AM

add

--local

--local

--local

--local when migrating

MMax (@rozenmd)add `--local` when migrating

A

a_sasin•3/22/24, 11:29 AM

But then the seed data migration will be applied to

—remote

—remote

—remote

—remote as well when deploying

Aa_sasin But then the seed data migration will be applied to `—remote` as well when deplo...

M

Max (@rozenmd)OP•3/22/24, 11:31 AM

I'm not sure I understand what you're trying to do here, no queries/migrations will be run against your remote DB unless you explicitly add the

--remote

--remote

--remote

--remote flag

E

Enduriel•3/22/24, 11:43 AM

say I have 100s or 1000s of rows I'd like to be able to insert during a query, but those rows are based on user input in such a way where I'd like to prevent sql injection, what's the proper way to handle this scenario?

should I just create a separate d1 statement for each row and bind them individually and then batch that all, is there a clever way to make this into one statement?

I want to create something equivalent to:

INSERT OR IGNORE INTO your_table_name (column1, column2, unique_column)
VALUES 
    (value1_1, value1_2, unique_value1),
    (value2_1, value2_2, unique_value2),
    (value3_1, value3_2, unique_value3),
    ...
    (valueN_1, valueN_2, unique_valueN);

INSERT OR IGNORE INTO your_table_name (column1, column2, unique_column)
VALUES 
    (value1_1, value1_2, unique_value1),
    (value2_1, value2_2, unique_value2),
    (value3_1, value3_2, unique_value3),
    ...
    (valueN_1, valueN_2, unique_valueN);

INSERT OR IGNORE INTO your_table_name (column1, column2, unique_column)
VALUES 
    (value1_1, value1_2, unique_value1),
    (value2_1, value2_2, unique_value2),
    (value3_1, value3_2, unique_value3),
    ...
    (valueN_1, valueN_2, unique_valueN);

INSERT OR IGNORE INTO your_table_name (column1, column2, unique_column)
VALUES 
    (value1_1, value1_2, unique_value1),
    (value2_1, value2_2, unique_value2),
    (value3_1, value3_2, unique_value3),
    ...
    (valueN_1, valueN_2, unique_valueN);

E

Enduriel•3/22/24, 11:47 AM

ah yeah

E

Enduriel•3/22/24, 11:47 AM

obviously

Original message was deleted

E

Enduriel•3/22/24, 11:47 AM

wow I feel kinda dumb lol, tysm

MMax (@rozenmd)an index is only useful if you're looking for a specific record/types of records...

V

vakno•3/22/24, 12:02 PM

i tried it but still i get same amount of rows read. i deleted the table and created the index. Am i missing something?

my index

'CREATE INDEX IF NOT EXISTS idx_url ON ProductData (url)'

'CREATE INDEX IF NOT EXISTS idx_url ON ProductData (url)'

'CREATE INDEX IF NOT EXISTS idx_url ON ProductData (url)'

'CREATE INDEX IF NOT EXISTS idx_url ON ProductData (url)'

my query

SELECT * FROM ProductData WHERE url = ?

SELECT * FROM ProductData WHERE url = ?

SELECT * FROM ProductData WHERE url = ?

SELECT * FROM ProductData WHERE url = ?

Vvakno i tried it but still i get same amount of rows read. i deleted the table and cre...

M

Max (@rozenmd)OP•3/22/24, 12:03 PM

how many rows is the table?

MMax (@rozenmd)how many rows is the table?

V

vakno•3/22/24, 12:03 PM

100

V

vakno•3/22/24, 12:03 PM

and i get 100 rows read in metrics

M

Max (@rozenmd)OP•3/22/24, 12:04 PM

and you're using an exact match, not a LIKE?

V

vakno•3/22/24, 12:04 PM

yes exact match

V

vakno•3/22/24, 12:09 PM

i run query like this

await env.productData.prepare('SELECT * FROM ProductData WHERE url = ?).bind(request.url).run();

await env.productData.prepare('SELECT * FROM ProductData WHERE url = ?).bind(request.url).run();

await env.productData.prepare('SELECT * FROM ProductData WHERE url = ?).bind(request.url).run();

await env.productData.prepare('SELECT * FROM ProductData WHERE url = ?).bind(request.url).run();

Original message was deleted

M

Max (@rozenmd)OP•3/22/24, 12:25 PM

my hero

V

vakno•3/22/24, 12:27 PM

yes but i deleted whole database and table. shouldnt that also delete the index?

V

vakno•3/22/24, 12:46 PM

found the issue, my function call for creating indexes was commented :NotLikeThis:

V

vakno•3/22/24, 12:46 PM

thanks for the help!

Vvakno found the issue, my function call for creating indexes was commented :NotLikeThi...

M

Matt Silverlock•3/22/24, 1:35 PM

You can validate if an index is used for a query - highly recommend reading this doc top to bottom! https://developers.cloudflare.com/d1/build-with-d1/use-indexes/#test-an-index

Cloudflare Docs

Use indexes · Cloudflare D1 docs

Indexes enable D1 to improve query performance over the indexed columns for common (popular) queries by reducing the amount of data (number of rows) …

A

Alsquiaga•3/22/24, 2:45 PM

When the D1 beta finishes, will the free tier remain active or will it be converted to paid feature?

AAlsquiaga When the D1 beta finishes, will the free tier remain active or will it be conver...

M

Max (@rozenmd)OP•3/22/24, 2:50 PM

The free tier is a core part of Cloudflare's business plan, and isn't going away

MMax (@rozenmd)The free tier is a core part of Cloudflare's business plan, and isn't going away

A

Alsquiaga•3/22/24, 2:51 PM

Thanks!

AAlsquiaga When the D1 beta finishes, will the free tier remain active or will it be conver...

M

Matt Silverlock•3/22/24, 2:58 PM

https://vxtwitter.com/elithrar/status/1765452226460340591?s=20

vxTwitter / fixvxTwitter

Matt Silverlock 🐀 (@elithrar)

We have no intentions to shelve or restrict the free tier of @cloudflare Workers, our D1 database, KV, or any of the other services we work to make accessible. Heck, we already have a couple of announcements that add to it.

(Apropos of nothing)

【QRT of Matt Silverlock

(@elithrar):】
'When we think about our “free” tier…

138

10

MMatt Silverlock https://vxtwitter.com/elithrar/status/1765452226460340591?s=20

A

Alsquiaga•3/22/24, 3:02 PM

Thanks a lot! I am very excited to start using these features for my projects!

M

Marcel Overdijk•3/22/24, 6:43 PM

Hi,

Recently I started experimenting with Cloudflare to host a website (static) and a dynamic REST API via workers.
Behind the REST API there is a database with approx. 200K rows which I need to periodically import into D1

So last week I did the initial import using Wrangler and it went really smoothly

A few minutes ago I tried to re-import and it failed with:

✘ [ERROR] Received a malformed response from the API

  <!DOCTYPE html>
  <!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
  <!--[if I... (length = 7300)

  POST
  /accounts/xxxxxxxxxxxxxxxxxxxxxxx/d1/database/xxxxxxxxxxxxxxxxxxxxxxx/query
  -> 520

  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

✘ [ERROR] Received a malformed response from the API

  <!DOCTYPE html>
  <!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
  <!--[if I... (length = 7300)

  POST
  /accounts/xxxxxxxxxxxxxxxxxxxxxxx/d1/database/xxxxxxxxxxxxxxxxxxxxxxx/query
  -> 520

  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

✘ [ERROR] Received a malformed response from the API

  <!DOCTYPE html>
  <!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
  <!--[if I... (length = 7300)

  POST
  /accounts/xxxxxxxxxxxxxxxxxxxxxxx/d1/database/xxxxxxxxxxxxxxxxxxxxxxx/query
  -> 520

  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

✘ [ERROR] Received a malformed response from the API

  <!DOCTYPE html>
  <!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
  <!--[if I... (length = 7300)

  POST
  /accounts/xxxxxxxxxxxxxxxxxxxxxxx/d1/database/xxxxxxxxxxxxxxxxxxxxxxx/query
  -> 520

  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

My import script first executes some

DROP TABLE IF EXISTS ..

DROP TABLE IF EXISTS ..

DROP TABLE IF EXISTS ..

DROP TABLE IF EXISTS .. statements, then creates the tables, and executes the inserts.
Although it failed, all tables were deleted, created and also some data was inserted.

It seems to have failed after approx. 25K rows out of 175K.

Are there any log files or something else I can check to see what happened?

Note I can import this file without any problem in a local sqlite database.

M

Marcel Overdijk•3/22/24, 6:45 PM

Another thing I now noticed is this in my D1 account page:

Rows read
1.25M

Rows written
375.35k

Rows read
1.25M

Rows written
375.35k

Rows read
1.25M

Rows written
375.35k

Rows read
1.25M

Rows written
375.35k

Before starting the import it said

0

0 and

0

0.
So I can not really explain these numbers as now only ~25K rows were inserted...

X

Xazu•3/22/24, 7:06 PM

Hi! I have a question. So how much data i can store on single db?

Screenshot_2024-03-22-20-02-29-080_com.android.chrome.jpg

X

Xazu•3/22/24, 7:08 PM

Is the limits page up-to-date?

C

Chaika•3/22/24, 7:08 PM

The first screenshot is showing pricing. On Paid, you get 5 GB included. The max DB size, as you see in the second screenshot, is different. 2 GB for paid per database.

XXazu Is the limits page up-to-date?

C

Chaika•3/22/24, 7:09 PM

yes, although they're working on bringing it up to 10 GB d1-database

C

Chaika•3/22/24, 7:09 PM

D1, as of right now, wants you to split your data among many databases rather then have one big one, thus the reasons of the limits being like that

X

Xazu•3/22/24, 7:10 PM

Ok. Thank you for the answer

S

sirhype•3/22/24, 9:35 PM

Can't figure out what I'm missing here. I've got two tables made. I'm using Drizzle ORM, and I'm trying to run some queries. But I get

D1_ERROR: no such table: inventory

D1_ERROR: no such table: inventory

D1_ERROR: no such table: inventory

D1_ERROR: no such table: inventory.

Any ideas where I might be going wrong? The statement it's trying to run is:

{
  sql: 'delete from "inventory" where ("inventory"."user_id" = ? and "inventory"."skin_id" not in (?, ?, ?, ?))',
  params: [
    '38891062',
    '9fa5664e-45fa-2a52-68e4-db97d85c8a91',
    '7e323e82-44b6-1711-2028-10af32c632d5',
    '19ba907b-4a76-347f-ea24-38990c9ff755',
    '7b71fc11-4fa2-c84c-31ec-5eb5c3e3fff3'
  ]
}

{
  sql: 'delete from "inventory" where ("inventory"."user_id" = ? and "inventory"."skin_id" not in (?, ?, ?, ?))',
  params: [
    '38891062',
    '9fa5664e-45fa-2a52-68e4-db97d85c8a91',
    '7e323e82-44b6-1711-2028-10af32c632d5',
    '19ba907b-4a76-347f-ea24-38990c9ff755',
    '7b71fc11-4fa2-c84c-31ec-5eb5c3e3fff3'
  ]
}

{
  sql: 'delete from "inventory" where ("inventory"."user_id" = ? and "inventory"."skin_id" not in (?, ?, ?, ?))',
  params: [
    '38891062',
    '9fa5664e-45fa-2a52-68e4-db97d85c8a91',
    '7e323e82-44b6-1711-2028-10af32c632d5',
    '19ba907b-4a76-347f-ea24-38990c9ff755',
    '7b71fc11-4fa2-c84c-31ec-5eb5c3e3fff3'
  ]
}

{
  sql: 'delete from "inventory" where ("inventory"."user_id" = ? and "inventory"."skin_id" not in (?, ?, ?, ?))',
  params: [
    '38891062',
    '9fa5664e-45fa-2a52-68e4-db97d85c8a91',
    '7e323e82-44b6-1711-2028-10af32c632d5',
    '19ba907b-4a76-347f-ea24-38990c9ff755',
    '7b71fc11-4fa2-c84c-31ec-5eb5c3e3fff3'
  ]
}

Creating the DB instance with:

  if (!env.DB) {
    throw new Error("Database connection not found");
  }
  const db = drizzle(env.DB, {
    schema,
    logger: true,
  });

  if (!env.DB) {
    throw new Error("Database connection not found");
  }
  const db = drizzle(env.DB, {
    schema,
    logger: true,
  });

  if (!env.DB) {
    throw new Error("Database connection not found");
  }
  const db = drizzle(env.DB, {
    schema,
    logger: true,
  });

  if (!env.DB) {
    throw new Error("Database connection not found");
  }
  const db = drizzle(env.DB, {
    schema,
    logger: true,
  });

Ssirhype Can't figure out what I'm missing here. I've got two tables made. I'm using Driz...

M

Max (@rozenmd)OP•3/22/24, 10:15 PM

bump wrangler to latest, explicitly use either

--local

--local

--local

--local or

--remote

--remote

--remote

--remote

J

Jacob Wright•3/22/24, 10:22 PM

Developer week is mid-May, right? Any hints for what we might get with D1? GA, multi-region replication, DurableObject-local Sqlite?

MMax (@rozenmd)bump wrangler to latest, explicitly use either `--local` or `--remote`

S

sirhype•3/22/24, 10:43 PM

ahhh the

--remote

--remote

--remote

--remote was what I was missing. Thank you!

T

Tony Leung•3/23/24, 10:47 AM

hi guys, is there any cache solution for D1 select query? With MySQL I can cache query by using ProxySQL.

P

PohKit•3/23/24, 5:01 PM

Hi guys,

Was exploring options between D1, PlanetScale & ManagedDB by Digital Ocean.

Just wanted to check has anyone tried with D1 x PHP/Laravel with this repo:
https://github.com/renoki-co/l1

I've created a Laravel project, written some basic API.

But seems like the Latency is high eg: 1,500ms ~ 2,000ms

Just wanted to confirm that, is D1 Laravel/PHP friendly?

GitHub

GitHub - renoki-co/l1: Extend your PHP/Laravel application with Clo...

Extend your PHP/Laravel application with Cloudflare bindings. - renoki-co/l1

TTony Leung hi guys, is there any cache solution for D1 select query? With MySQL I can cache...

S

squareclamp•3/23/24, 11:34 PM

Not sure if there is anything out of the box, but you can look at the cache api - https://developers.cloudflare.com/workers/runtime-apis/cache/

Cloudflare Docs

Cache · Cloudflare Workers docs

Control reading and writing from the Cloudflare global network cache.

X

x03•3/24/24, 4:12 AM

How do I apply migrations on my production database? To apply local migrations, I just run:

bun wrangler d1 migrations apply db_preview --local

bun wrangler d1 migrations apply db_preview --local

bun wrangler d1 migrations apply db_preview --local

bun wrangler d1 migrations apply db_preview --local

This is a SvelteKit pages app.

X

x03•3/24/24, 4:13 AM

And my wrangler.toml if needed:

[[d1_databases]]
binding = "DB"
database_name = "db_preview"
database_id = "db_preview"
migrations_dir = "./drizzle"

[[d1_databases]]
binding = "DB"
database_name = "db_preview"
database_id = "db_preview"
migrations_dir = "./drizzle"

[[d1_databases]]
binding = "DB"
database_name = "db_preview"
database_id = "db_preview"
migrations_dir = "./drizzle"

[[d1_databases]]
binding = "DB"
database_name = "db_preview"
database_id = "db_preview"
migrations_dir = "./drizzle"

Xx03 How do I apply migrations on my production database? To apply local migrations, ...

P

predaytor•3/24/24, 9:49 AM

there is

--remote

--remote

--remote

--remote flag

a single DO instance can do around 1k requests per second, for complicated queries i've seen it drop

Similar Threads

Similar Threads

Similar Threads