is there a limit to the size of queries you can run with `wrangler d1 execute`?

Mmicromashor is there a limit to the size of queries you can run with `wrangler d1 execute`?

U

Unsmart•7/3/23, 3:23 PM

yes: https://developers.cloudflare.com/d1/platform/limits/

Many of these limits will increase during D1’s public alpha. Join the #d1-open-alpha channel in the Cloudflare Developer Discord to keep up to date …

M

micromashorOP•7/3/23, 3:23 PM

hmmmm, I've managed to lock up the dashboard console with a large query still well under 100k, probably just the browser not being able to handle pasting that much data into a single input field lol

Vvvarp hey, I am researching on how well D1 suits my use case, and I wonder if it will ...

U

Unsmart•7/3/23, 9:15 PM

d1-database

CCole Xemi I see that there is now metrics for D1 yet I am not seeing the metrics tab for m...

A

Adi•7/4/23, 1:22 AM

It's possible that your database is using the old backend? My understanding is that metrics are currently only available for databases using the new experimental backend for D1.

Mmicromashor hmmmm, I've managed to lock up the dashboard console with a large query still we...

M

Max (@rozenmd)•7/4/23, 8:40 AM

i've pasted multi-MB files into the text field and had them go through (the UI completely crashed, but the query went through lul

)

M

Max (@rozenmd)•7/4/23, 8:40 AM

but yeah it's not (yet) designed for large amounts of data

B

bkyerv•7/4/23, 8:57 AM

hello everyone. is there any way to check if connection to d1 is successful? I am facing an error that the table I am querying doesn't exist but in the ui dashboard I see the table (i.e. it does exist) now I want to see if connection has been properly established. what is the best way to check it?

Bbkyerv hello everyone. is there any way to check if connection to d1 is successful? I a...

M

Max (@rozenmd)•7/4/23, 10:14 AM

you're likely talking to D1 locally rather than remotely, see https://developers.cloudflare.com/d1/learning/local-development/#start-a-local-development-session

MMax (@rozenmd)you're likely talking to D1 locally rather than remotely, see https://developers...

B

bkyerv•7/4/23, 11:06 AM

thank you. that was the reason

C

Colin•7/4/23, 3:45 PM

Hi Everyone. I'm new to this Discord. I am planning to use D1 for my new website. Cloudflare says they don't recommend it for production. I will have a simple interface (about 5 tables). I will be maintaining data about my clients permission to login, some client data stored as JSON strings, and some other basic stuff. Should I be concerned about the stabilithy of D1 in a production website?

U

Unsmart•7/4/23, 3:47 PM

It is an alpha product so breaking changes can happen and while it probably will work fine you probably shouldnt use it for production.

TTechist Created a D1 database via Web and created bindings. Unable to write to it, fro...

M

Matt Silverlock•7/4/23, 7:50 PM

If you are using Pages, have you followed the docs? https://developers.cloudflare.com/pages/platform/functions/bindings/#d1-databases

Bindings · Cloudflare Pages docs

A binding enables your Pages Functions to interact with resources on the Cloudflare developer platform. Use bindings to integrate your Pages Functions …

TTechist Created a D1 database via Web and created bindings. Unable to write to it, fro...

M

Matt Silverlock•7/4/23, 7:51 PM

It will not be on

env

env

env

env but instead

context.env

context.env

context.env

context.env per the linked docs

M

Matt Silverlock•7/4/23, 7:51 PM

interface Env {
  YOUR_DB: D1Database;
}

export const onRequest: PagesFunction<Env> = async (context) => {
  // Create a prepared statement with our query
  const ps = context.env.YOUR_DB.prepare('SELECT * from users');
  const data = await ps.first();

  return Response.json(data);
}

interface Env {
  YOUR_DB: D1Database;
}

export const onRequest: PagesFunction<Env> = async (context) => {
  // Create a prepared statement with our query
  const ps = context.env.YOUR_DB.prepare('SELECT * from users');
  const data = await ps.first();

  return Response.json(data);
}

interface Env {
  YOUR_DB: D1Database;
}

export const onRequest: PagesFunction<Env> = async (context) => {
  // Create a prepared statement with our query
  const ps = context.env.YOUR_DB.prepare('SELECT * from users');
  const data = await ps.first();

  return Response.json(data);
}

interface Env {
  YOUR_DB: D1Database;
}

export const onRequest: PagesFunction<Env> = async (context) => {
  // Create a prepared statement with our query
  const ps = context.env.YOUR_DB.prepare('SELECT * from users');
  const data = await ps.first();

  return Response.json(data);
}

TTechist Do I need to have this? Haven't seen this in the documentation. interface En...

M

Matt Silverlock•7/4/23, 8:33 PM

If you use TypeScript, yes. Read the docs there - follow each step! And show your code

C

Catt0s•7/4/23, 9:29 PM

Not sure what I'm doing wrong, and I'm not experienced with JS or SQL, but
1) the console works fine
2) invalid query throw errors
but...
3) valid queries return nothing?

console.log(query);
result = context.env.db.prepare(query).bind(value);
const res = await result.raw();
console.log("RESULT: "+res);

console.log(query);
result = context.env.db.prepare(query).bind(value);
const res = await result.raw();
console.log("RESULT: "+res);

console.log(query);
result = context.env.db.prepare(query).bind(value);
const res = await result.raw();
console.log("RESULT: "+res);

console.log(query);
result = context.env.db.prepare(query).bind(value);
const res = await result.raw();
console.log("RESULT: "+res);

p.s. - please leave ping on for replies (ping me after if your forget)

CCatt0s Not sure what I'm doing wrong, and I'm not experienced with JS or SQL, but 1) t...

M

Matt Silverlock•7/4/23, 11:51 PM

Share all of your code. Does your query return anything at all? Are you sure? If your bound values are null they may not match any rows.

CCatt0s Not sure what I'm doing wrong, and I'm not experienced with JS or SQL, but 1) t...

M

Matt Silverlock•7/4/23, 11:52 PM

You should log your values BEFORE you run the query to ensure they are not null

MMatt Silverlock You should log your values BEFORE you run the query to ensure they are not null

C

Catt0s•7/5/23, 2:29 AM

can you explain what you mean? I am only using static test data for now, and the query I am running (which i log to console as well) works fine in the D1 console.
logged query:

SELECT * from items where id=(SELECT item_id FROM parts WHERE ? in (name, id))

SELECT * from items where id=(SELECT item_id FROM parts WHERE ? in (name, id))

SELECT * from items where id=(SELECT item_id FROM parts WHERE ? in (name, id))

SELECT * from items where id=(SELECT item_id FROM parts WHERE ? in (name, id))
binded

?

? to

0

0. Here it is in console:

MMatt Silverlock Share all of your code. Does your query return anything at all? Are you sure? If...

C

Catt0s•7/5/23, 2:46 AM

I was creating this minimum code to reproduce my issue:

export async function onRequestGet(context) {
  const value = 0; // normally this is from the URL
  var partialQuery = "SELECT item_id FROM ";
  partialQuery+= 'categories';
  partialQuery+=" WHERE ? in (name, id)";
  var query = "SELECT * from items where id=("+partialQuery+")";
  console.log(query);
  var result = context.env.DB.
    prepare(query).bind(value);
  const res = await result.raw();
  console.log("RESULT: "+res);
  return new Response("");
}

export async function onRequestGet(context) {
  const value = 0; // normally this is from the URL
  var partialQuery = "SELECT item_id FROM ";
  partialQuery+= 'categories';
  partialQuery+=" WHERE ? in (name, id)";
  var query = "SELECT * from items where id=("+partialQuery+")";
  console.log(query);
  var result = context.env.DB.
    prepare(query).bind(value);
  const res = await result.raw();
  console.log("RESULT: "+res);
  return new Response("");
}

export async function onRequestGet(context) {
  const value = 0; // normally this is from the URL
  var partialQuery = "SELECT item_id FROM ";
  partialQuery+= 'categories';
  partialQuery+=" WHERE ? in (name, id)";
  var query = "SELECT * from items where id=("+partialQuery+")";
  console.log(query);
  var result = context.env.DB.
    prepare(query).bind(value);
  const res = await result.raw();
  console.log("RESULT: "+res);
  return new Response("");
}

export async function onRequestGet(context) {
  const value = 0; // normally this is from the URL
  var partialQuery = "SELECT item_id FROM ";
  partialQuery+= 'categories';
  partialQuery+=" WHERE ? in (name, id)";
  var query = "SELECT * from items where id=("+partialQuery+")";
  console.log(query);
  var result = context.env.DB.
    prepare(query).bind(value);
  const res = await result.raw();
  console.log("RESULT: "+res);
  return new Response("");
}

and in doing so, I have found the problem(s)/possible bugs(? I do not know as I am not experienced with databases):

When getting parameters from the URL they are lowercase and are strings.

I was under the impression due to https://developers.cloudflare.com/d1/platform/client-api/#type-conversion that it would be converted for me. So, I need to solve the following problems:
1) Numbers in my url need to be changed from

strings

strings

strings

strings to

int

int

s.
2) I need to make my query non case-sensitive

C

Catt0s•7/5/23, 2:47 AM

in other words, my

value

value

value

value was

'0'

'0'

'0'

'0' and not

0

0 and my

'Test_Entry'

'Test_Entry'

'Test_Entry'

'Test_Entry' was

'test_entry'

'test_entry'

'test_entry'

'test_entry' which is why nothing was working

B

bkyerv•7/5/23, 8:52 AM

I can clearly list all production d1 dbs by running

wrangler d1 list

wrangler d1 list

wrangler d1 list

wrangler d1 list. is there similar way to detect what local d1 dbs are available?

B

bkyerv•7/5/23, 8:54 AM

This is important because when running into errors it helps a lot to first check if the local db has been properly set and if the migration ran successfully against the db (i.e. being able to check the contents of the db). In other words it would help to check potential causes of the error when running queries

B

bkyerv•7/5/23, 9:03 AM

To enhance the user experience when working with different environments in our local development setup, it would be cool if an api has been similar to Python's environment management. For instance, we could introduce a command like

wrangler source dev

wrangler source dev

wrangler source dev

wrangler source dev. Upon execution, this command would switch to the development environment. As a result, any subsequent command using wrangler would interact with local development databases and settings. This approach could simplify the development workflow by eliminating potential confusions related to various flags such as --persist, --local, and others.

B

bkyerv•7/5/23, 9:04 AM

this is just speaking out my ideas/desire and not a complain or criticism

B

bkyerv•7/5/23, 9:19 AM

running

wrangler d1 execute <database-name> --local --file ./schema.sql

wrangler d1 execute <database-name> --local --file ./schema.sql

wrangler d1 execute <database-name> --local --file ./schema.sql

wrangler d1 execute <database-name> --local --file ./schema.sql command creates a db in the

.wrangler

.wrangler

.wrangler

.wrangler folder. Is it possible to set a different location to put the db to?

B

bkyerv•7/5/23, 9:22 AM

turns out yes it is possible to set the location by adding --persist-to flag to the command above. Now when I figured it it seems obvious, but when reading the docs it is not that obvious you can add flags like --persist-to not only to the wrangler dev command as shown in the docs as an example but to other commands as well

MMatt Silverlock We have some ideas here for the “thousands to tens of thousands of tenants” - th...

G

GuillaumeLakano•7/5/23, 1:24 PM

If D1 could have unlimited databases like we have unlimited DO (and D1 is based on DO...) this could permit us to add a very strong security layer by associate a dedicated D1 DB to each user, but this is not possible if it's limited (even « thousands to tens of thousands of tenants » )

GGuillaumeLakano If D1 could have unlimited databases like we have unlimited DO (and D1 is based ...

I

itsmatteomanf•7/5/23, 3:13 PM

Do note that DOs will, eventually, expose their own private SQL instance, with the same interface (or similar, at least) as D1. Which might solve your problem

Ssebosp Hello! Does anybody know if something is going on? I suddenly start receiving th...

C

Cyb3r-Jak3•7/5/23, 4:31 PM

pages-discussions

Ssebosp thank you, what is this reference for?

C

Cyb3r-Jak3•7/5/23, 4:40 PM

There is an incident going on with workers which could be causing the error code

Jjschlesser Is there a way to see what the cost of a given D1 query is, in terms of billable...

M

Matt Silverlock•7/5/23, 10:22 PM

Not yet, but we will (before we start billing) instrument responses with this info. You should expect to see readUnits and/or writeUnits for any given query or batch.

Jjschlesser Is there a way to see what the cost of a given D1 query is, in terms of billable...

M

Matt Silverlock•7/5/23, 10:26 PM

Reads & writes primarily map to underlying pages. A full table scan of (e.g) 100k rows @ 200B each = ~5,000 read units. A single row lookup on an indexed column would typically be 2 units (one for the index, one for the row itself).