Any way to improve performance, it seems all queries takes 1.xx seconds, even on Cloudflare dashboar

VVikash Any way to improve performance, it seems all queries takes 1.xx seconds, even on...

I

If all queries are slow can you check if you're on the beta or alpha version, and also whether anything else is slow other than D1? (for example, D1 is slow on my internet if I have an ongoing backup, since it saturates my internet link and slows everything else down)

VVikash Any way to improve performance, it seems all queries takes 1.xx seconds, even on...

M

Max (@rozenmd)•10/4/23, 11:41 AM

the speed of the API in the dashboard isn't indicative of what you should be seeing from your Worker - deploy a Worker and see how long your requests are taking

IIsaac McFadyen If all queries are slow can you check if you're on the beta or alpha version, an...

V

VikashOP•10/4/23, 11:46 AM

Only D1 is slow, I am using other options like KV which works fine. How to check alpha or beta? I created the database via dashboard 2 weeks ago.

VVikash Only D1 is slow, I am using other options like KV which works fine. How to check...

I

Isaac McFadyen•10/4/23, 11:47 AM

If you created it that recently it should be

beta

beta

, yes.

I

Isaac McFadyen•10/4/23, 11:47 AM

Alpha is the older D1 engine that doesn't work as well and is deprecated.

MMax (@rozenmd)the speed of the API in the dashboard isn't indicative of what you should be see...

V

VikashOP•10/4/23, 11:47 AM

I deployed the worker and getting same result -

V

VikashOP•10/4/23, 11:49 AM

While the KV is returning in less then 50ms

VVikash I deployed the worker and getting same result -

M

Max (@rozenmd)•10/4/23, 12:19 PM

what's the output of

wrangler d1 info <dbname>

wrangler d1 info <dbname>

?

MMax (@rozenmd)what's the output of `wrangler d1 info <dbname>`?

V

VikashOP•10/4/23, 12:22 PM

VVikash Click to see attachment

I

Isaac McFadyen•10/4/23, 1:58 PM

Are you located in APAC (Asia-Pacific)?

IIsaac McFadyen Are you located in APAC (Asia-Pacific)?

V

VikashOP•10/4/23, 2:02 PM

Yes, India

EEd Does this endpoint still work: `https://api.cloudflare.com/client/v4/accounts/${...

C

Cyb3r-Jak3•10/4/23, 5:05 PM

CORS is explicitly denied for api.cloudflare.com. You need else to proxy the request.

C

cfanclub•10/4/23, 5:29 PM

I want a help to use local sqlite created under .wrangler folder for local development and apply migration.

I used the command of

wrangler d1 migrations testdb apply --local

wrangler d1 migrations testdb apply --local

command to apply migration, it worked successfully but when I use the prepare statement I am getting error of no table Test exists

G

God•10/4/23, 5:34 PM

Does D1 have any builtin support for generating an UUID? or is that something I should handle on the JS side?

GGod Does D1 have any builtin support for generating an UUID? or is that something I ...

I

Isaac McFadyen•10/4/23, 5:34 PM

JS-side, probably yeah.

C

Cretezy•10/4/23, 5:46 PM

Hi, congratulations on the open beta!
I would like to propose a feature: Exporting databases.
It's important to be able to do external backups, as well as be able to download entire databases for local analysis. Thank you.

CCretezy Hi, congratulations on the open beta! I would like to propose a feature: Exporti...

I

itsmatteomanf•10/4/23, 8:22 PM

There are packages that do it...

https://github.com/nora-soderlund/cloudflare-d1-backups

GitHub

GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a ...

A package to dump a copy of your experimental D1 database through a worker. - GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a copy of your experimental D1 database through a worker.

CCretezy Hi, congratulations on the open beta! I would like to propose a feature: Exporti...

M

Max (@rozenmd)•10/4/23, 8:39 PM

it's on my mind to add that, yep

VVikash Yes, India

V

VikashOP•10/5/23, 2:50 AM

The database is for testing, feel free to takeover if you want to test? Or should I create new to test in different region?

O

Old Man Maker•10/5/23, 3:37 AM

Hi - I started getting these errors in a call that previously was not having errors:

Object {
  message: D1_ERROR: Failed to parse body as JSON, got: error code: 1042,
  cause: Failed to parse body as JSON, got: error code: 1042                                        
}

Object {
  message: D1_ERROR: Failed to parse body as JSON, got: error code: 1042,
  cause: Failed to parse body as JSON, got: error code: 1042                                        
}

Object {
  message: D1_ERROR: Failed to parse body as JSON, got: error code: 1042,
  cause: Failed to parse body as JSON, got: error code: 1042                                        
}

Object {
  message: D1_ERROR: Failed to parse body as JSON, got: error code: 1042,
  cause: Failed to parse body as JSON, got: error code: 1042                                        
}

Having checked the variables are all correct and the batching syntax appears correct, I'm a bit stuck on how to further debug this

OOld Man Maker Hi - I started getting these errors in a call that previously was not having err...

U

Unsmart•10/5/23, 3:40 AM

Can you share the code where you get this?

UUnsmart Can you share the code where you get this?

O

Old Man Maker•10/5/23, 3:42 AM

This and other functions are all now failing with the same error.
The code is

async function InsertIntoContentDatabase(db:D1Database, content:ContentDataType): Promise<D1Result<unknown>[]>{

    try {


        if(content.contentDataItemType === undefined){
            console.log('No contentDataItemType');            
            throw('Missing contentDataItemType Array');
        }


        //Because we may have multiple items to push into the database, we need to batch the call
        const batcher = await db.prepare("INSERT INTO content_models (contentGroupRef, projectRef, contentElementRef, contentBakedStatement, publishState) VALUES (?1, ?2, ?3, ?4, ?5)")

        let binderCollection=[];

        for (let index = 0; index < content.contentDataItemType.length; index++) {
            const element = content.contentDataItemType[index];

            let temp = batcher.bind(
                element.contentGroupRef, 
                element.projectRef,
                element.contentElementRef,
                element.contentBakedStatement,
                element.publishState
            );
            binderCollection.push(temp);                

            console.log('Loop Batch=>', JSON.stringify(temp));
        }


        return await db.batch(binderCollection);
        
        
    } catch (error:any) {

        console.log('D1 Instance=>', JSON.stringify(db.dump()));

        console.log({
            message: error.message,
            cause: error.cause.message,
        });
        throw(error);
    }
}

async function InsertIntoContentDatabase(db:D1Database, content:ContentDataType): Promise<D1Result<unknown>[]>{

    try {


        if(content.contentDataItemType === undefined){
            console.log('No contentDataItemType');            
            throw('Missing contentDataItemType Array');
        }


        //Because we may have multiple items to push into the database, we need to batch the call
        const batcher = await db.prepare("INSERT INTO content_models (contentGroupRef, projectRef, contentElementRef, contentBakedStatement, publishState) VALUES (?1, ?2, ?3, ?4, ?5)")

        let binderCollection=[];

        for (let index = 0; index < content.contentDataItemType.length; index++) {
            const element = content.contentDataItemType[index];

            let temp = batcher.bind(
                element.contentGroupRef, 
                element.projectRef,
                element.contentElementRef,
                element.contentBakedStatement,
                element.publishState
            );
            binderCollection.push(temp);                

            console.log('Loop Batch=>', JSON.stringify(temp));
        }


        return await db.batch(binderCollection);
        
        
    } catch (error:any) {

        console.log('D1 Instance=>', JSON.stringify(db.dump()));

        console.log({
            message: error.message,
            cause: error.cause.message,
        });
        throw(error);
    }
}

async function InsertIntoContentDatabase(db:D1Database, content:ContentDataType): Promise<D1Result<unknown>[]>{

    try {


        if(content.contentDataItemType === undefined){
            console.log('No contentDataItemType');            
            throw('Missing contentDataItemType Array');
        }


        //Because we may have multiple items to push into the database, we need to batch the call
        const batcher = await db.prepare("INSERT INTO content_models (contentGroupRef, projectRef, contentElementRef, contentBakedStatement, publishState) VALUES (?1, ?2, ?3, ?4, ?5)")

        let binderCollection=[];

        for (let index = 0; index < content.contentDataItemType.length; index++) {
            const element = content.contentDataItemType[index];

            let temp = batcher.bind(
                element.contentGroupRef, 
                element.projectRef,
                element.contentElementRef,
                element.contentBakedStatement,
                element.publishState
            );
            binderCollection.push(temp);                

            console.log('Loop Batch=>', JSON.stringify(temp));
        }


        return await db.batch(binderCollection);
        
        
    } catch (error:any) {

        console.log('D1 Instance=>', JSON.stringify(db.dump()));

        console.log({
            message: error.message,
            cause: error.cause.message,
        });
        throw(error);
    }
}

async function InsertIntoContentDatabase(db:D1Database, content:ContentDataType): Promise<D1Result<unknown>[]>{

    try {


        if(content.contentDataItemType === undefined){
            console.log('No contentDataItemType');            
            throw('Missing contentDataItemType Array');
        }


        //Because we may have multiple items to push into the database, we need to batch the call
        const batcher = await db.prepare("INSERT INTO content_models (contentGroupRef, projectRef, contentElementRef, contentBakedStatement, publishState) VALUES (?1, ?2, ?3, ?4, ?5)")

        let binderCollection=[];

        for (let index = 0; index < content.contentDataItemType.length; index++) {
            const element = content.contentDataItemType[index];

            let temp = batcher.bind(
                element.contentGroupRef, 
                element.projectRef,
                element.contentElementRef,
                element.contentBakedStatement,
                element.publishState
            );
            binderCollection.push(temp);                

            console.log('Loop Batch=>', JSON.stringify(temp));
        }


        return await db.batch(binderCollection);
        
        
    } catch (error:any) {

        console.log('D1 Instance=>', JSON.stringify(db.dump()));

        console.log({
            message: error.message,
            cause: error.cause.message,
        });
        throw(error);
    }
}

O

Old Man Maker•10/5/23, 3:42 AM

each loop batch has this being pushed:

Loop Batch=> {"database":{"fetcher":{}},"statement":"UPDATE content_models SET contentGroupRef = ?2, projectRef = ?3, contentBakedStatement = ?4, publishState = ?5 WHERE contentElementRef = ?1","params":["ref0","cg0","prTest0","We are proud to present a gorgeous large green wall panel square made from ivory wood.",1]}

Loop Batch=> {"database":{"fetcher":{}},"statement":"UPDATE content_models SET contentGroupRef = ?2, projectRef = ?3, contentBakedStatement = ?4, publishState = ?5 WHERE contentElementRef = ?1","params":["ref0","cg0","prTest0","We are proud to present a gorgeous large green wall panel square made from ivory wood.",1]}

Loop Batch=> {"database":{"fetcher":{}},"statement":"UPDATE content_models SET contentGroupRef = ?2, projectRef = ?3, contentBakedStatement = ?4, publishState = ?5 WHERE contentElementRef = ?1","params":["ref0","cg0","prTest0","We are proud to present a gorgeous large green wall panel square made from ivory wood.",1]}

Loop Batch=> {"database":{"fetcher":{}},"statement":"UPDATE content_models SET contentGroupRef = ?2, projectRef = ?3, contentBakedStatement = ?4, publishState = ?5 WHERE contentElementRef = ?1","params":["ref0","cg0","prTest0","We are proud to present a gorgeous large green wall panel square made from ivory wood.",1]}

O

Old Man Maker•10/5/23, 3:43 AM

thats update. this is insert:

Loop Batch=> {"database":{"fetcher":{}},"statement":"INSERT INTO content_models (contentGroupRef, projectRef, contentElementRef, contentBakedStatement, publishState) VALUES (?1, ?2, ?3, ?4, ?5)","params":["cg0","prTest0","ref0","We are proud to present a gorgeous large green wall panel square made from ivory wood.",1]}

Loop Batch=> {"database":{"fetcher":{}},"statement":"INSERT INTO content_models (contentGroupRef, projectRef, contentElementRef, contentBakedStatement, publishState) VALUES (?1, ?2, ?3, ?4, ?5)","params":["cg0","prTest0","ref0","We are proud to present a gorgeous large green wall panel square made from ivory wood.",1]}

Loop Batch=> {"database":{"fetcher":{}},"statement":"INSERT INTO content_models (contentGroupRef, projectRef, contentElementRef, contentBakedStatement, publishState) VALUES (?1, ?2, ?3, ?4, ?5)","params":["cg0","prTest0","ref0","We are proud to present a gorgeous large green wall panel square made from ivory wood.",1]}

Loop Batch=> {"database":{"fetcher":{}},"statement":"INSERT INTO content_models (contentGroupRef, projectRef, contentElementRef, contentBakedStatement, publishState) VALUES (?1, ?2, ?3, ?4, ?5)","params":["cg0","prTest0","ref0","We are proud to present a gorgeous large green wall panel square made from ivory wood.",1]}

O

Old Man Maker•10/5/23, 3:44 AM

The databases fetcher property is showing {}

U

Unsmart•10/5/23, 3:44 AM

the fetcher logging as {} is expected afaik... but can you see if the operations actually go through? This does seem like its probably an internal error but wonder if its when trying to parse the request or the response

O

Old Man Maker•10/5/23, 3:45 AM

they do not go through. Just created another duplicate D1 instance with no data to see if the insert works. No dice

U

Unsmart•10/5/23, 3:46 AM

Hmm yeah thats weird does a normal non batch operation work?

O

Old Man Maker•10/5/23, 3:46 AM

I'll give it shot BRB

O

Old Man Maker•10/5/23, 3:50 AM

No - it failed as well.

Error=> Error: D1_ERROR: Failed to parse body as JSON, got: error code: 1042                                                                                                                                                                                                                                                                                        
    at D1Database._send (cloudflare-internal:d1-api:105:19)                                                                                                                                                                                                                                                                                                         
    at async D1Database.exec (cloudflare-internal:d1-api:41:23)                                                                                                                                                                                                                                                                                                     
    at async InsertIntoContentDatabase (file:///C:XXXXX/Temp/tmp-35236-4jsc2tuBSi4U/index.js:3232:5)
    at async RunContent (file:///C:XXXXX/Temp/tmp-35236-4jsc2tuBSi4U/index.js:3155:17)
    at async Array.<anonymous> (file:///C:XXXXX/Temp/tmp-35236-4jsc2tuBSi4U/index.js:3727:12)
    at async file:///C:XXXXX/Temp/tmp-35236-4jsc2tuBSi4U/index.js:2408:21 {
  stack: Error: D1_ERROR: Failed to parse body as JSON, got…ocal/Temp/tmp-35236-4jsc2tuBSi4U/index.js:2408:21,
  message: D1_ERROR: Failed to parse body as JSON, got: error code: 1042,
  cause: Error: Failed to parse body as JSON, got: error code: 1042
    at D1Database._send (cloudflare-inte…
}

Error=> Error: D1_ERROR: Failed to parse body as JSON, got: error code: 1042                                                                                                                                                                                                                                                                                        
    at D1Database._send (cloudflare-internal:d1-api:105:19)                                                                                                                                                                                                                                                                                                         
    at async D1Database.exec (cloudflare-internal:d1-api:41:23)                                                                                                                                                                                                                                                                                                     
    at async InsertIntoContentDatabase (file:///C:XXXXX/Temp/tmp-35236-4jsc2tuBSi4U/index.js:3232:5)
    at async RunContent (file:///C:XXXXX/Temp/tmp-35236-4jsc2tuBSi4U/index.js:3155:17)
    at async Array.<anonymous> (file:///C:XXXXX/Temp/tmp-35236-4jsc2tuBSi4U/index.js:3727:12)
    at async file:///C:XXXXX/Temp/tmp-35236-4jsc2tuBSi4U/index.js:2408:21 {
  stack: Error: D1_ERROR: Failed to parse body as JSON, got…ocal/Temp/tmp-35236-4jsc2tuBSi4U/index.js:2408:21,
  message: D1_ERROR: Failed to parse body as JSON, got: error code: 1042,
  cause: Error: Failed to parse body as JSON, got: error code: 1042
    at D1Database._send (cloudflare-inte…
}

Error=> Error: D1_ERROR: Failed to parse body as JSON, got: error code: 1042                                                                                                                                                                                                                                                                                        
    at D1Database._send (cloudflare-internal:d1-api:105:19)                                                                                                                                                                                                                                                                                                         
    at async D1Database.exec (cloudflare-internal:d1-api:41:23)                                                                                                                                                                                                                                                                                                     
    at async InsertIntoContentDatabase (file:///C:XXXXX/Temp/tmp-35236-4jsc2tuBSi4U/index.js:3232:5)
    at async RunContent (file:///C:XXXXX/Temp/tmp-35236-4jsc2tuBSi4U/index.js:3155:17)
    at async Array.<anonymous> (file:///C:XXXXX/Temp/tmp-35236-4jsc2tuBSi4U/index.js:3727:12)
    at async file:///C:XXXXX/Temp/tmp-35236-4jsc2tuBSi4U/index.js:2408:21 {
  stack: Error: D1_ERROR: Failed to parse body as JSON, got…ocal/Temp/tmp-35236-4jsc2tuBSi4U/index.js:2408:21,
  message: D1_ERROR: Failed to parse body as JSON, got: error code: 1042,
  cause: Error: Failed to parse body as JSON, got: error code: 1042
    at D1Database._send (cloudflare-inte…
}

Error=> Error: D1_ERROR: Failed to parse body as JSON, got: error code: 1042                                                                                                                                                                                                                                                                                        
    at D1Database._send (cloudflare-internal:d1-api:105:19)                                                                                                                                                                                                                                                                                                         
    at async D1Database.exec (cloudflare-internal:d1-api:41:23)                                                                                                                                                                                                                                                                                                     
    at async InsertIntoContentDatabase (file:///C:XXXXX/Temp/tmp-35236-4jsc2tuBSi4U/index.js:3232:5)
    at async RunContent (file:///C:XXXXX/Temp/tmp-35236-4jsc2tuBSi4U/index.js:3155:17)
    at async Array.<anonymous> (file:///C:XXXXX/Temp/tmp-35236-4jsc2tuBSi4U/index.js:3727:12)
    at async file:///C:XXXXX/Temp/tmp-35236-4jsc2tuBSi4U/index.js:2408:21 {
  stack: Error: D1_ERROR: Failed to parse body as JSON, got…ocal/Temp/tmp-35236-4jsc2tuBSi4U/index.js:2408:21,
  message: D1_ERROR: Failed to parse body as JSON, got: error code: 1042,
  cause: Error: Failed to parse body as JSON, got: error code: 1042
    at D1Database._send (cloudflare-inte…
}

O

Old Man Maker•10/5/23, 3:51 AM

funny that it fails with exactly the same error code

U

Unsmart•10/5/23, 3:53 AM

Hmm thats really weird... what wrangler version are you on?

O

Old Man Maker•10/5/23, 3:55 AM

3.10.0

O

Old Man Maker•10/5/23, 3:56 AM

i've also just started getting issues with vectorize at the same time. Appears that something bad has happened in the local version.

I did deploy the worker and verified that the deploy works - so we are local issue only here at the moment.

O

Old Man Maker•10/5/23, 3:56 AM

I restarted my machine, tried to clear out caches etc.

O

Old Man Maker•10/5/23, 3:56 AM

any way to force that through CLI to refresh it all/

U

Unsmart•10/5/23, 3:57 AM

Ah ok this is local good to know... I was gonna say the one thing I have deployed seems to work fine.

U

Unsmart•10/5/23, 4:00 AM

I do not use D1 really at all though and the error seems weird I'd probably just create an issue on the workers-sdk repo about it and maybe try an older version to see if it works or even remote mode

https://github.com/cloudflare/workers-sdk

O

Old Man Maker•10/5/23, 4:00 AM

i'm running in remote mode for the vectorize side of it

U

Unsmart•10/5/23, 4:01 AM

ah ok if you run in local mode does D1 work?

O

Old Man Maker•10/5/23, 4:01 AM

i'll try, but I think everything else fails then. Has been working in remote mode for a few days - no changes to those lines of code in the functions

U

Unsmart•10/5/23, 4:02 AM

Yeah trying to see if this is specific to remote mode

O

Old Man Maker•10/5/23, 4:04 AM

hmmm... now that's a really strange error:

Object {
  message: D1_EXEC_ERROR: Error in line 1: INSERT INTO conten…'test1','test1',1): no such table: content_models,
  cause: Error in line 1: INSERT INTO content_models (conte…'test1','test1',1): no such table: content_models
}

Object {
  message: D1_EXEC_ERROR: Error in line 1: INSERT INTO conten…'test1','test1',1): no such table: content_models,
  cause: Error in line 1: INSERT INTO content_models (conte…'test1','test1',1): no such table: content_models
}

Object {
  message: D1_EXEC_ERROR: Error in line 1: INSERT INTO conten…'test1','test1',1): no such table: content_models,
  cause: Error in line 1: INSERT INTO content_models (conte…'test1','test1',1): no such table: content_models
}

Object {
  message: D1_EXEC_ERROR: Error in line 1: INSERT INTO conten…'test1','test1',1): no such table: content_models,
  cause: Error in line 1: INSERT INTO content_models (conte…'test1','test1',1): no such table: content_models
}

The table certainly does exist both in dashboard and on list

O

Old Man Maker•10/5/23, 4:04 AM

So these appear to be binding issues with Wrangler

U

Unsmart•10/5/23, 4:05 AM

Well in local mode it would not exist because it wont pull any table info from ur actual db

O

Old Man Maker•10/5/23, 4:06 AM

ok - I'm confused

U

Unsmart•10/5/23, 4:07 AM

In local mode it is fully locally and will not know anything about ur cloudflare account. So you would basically need to create the table locally for it to exist

Any way to improve performance, it seems all queries takes 1.xx seconds, even on Cloudflare dashboar

Similar Threads

Similar Threads

Similar Threads