I need to do 10k fetches to api end point. Responses can take up to 15 seconds. What would be the co

AAdvany I need to do 10k fetches to api end point. Responses can take up to 15 seconds. ...

S

sathoro•5/31/23, 10:06 AM

using DO as a queue should work well. you would need to do the pricing calculations though. I think we are talking less than a few dollars though (probably much less)...

Z

zegevlier•5/31/23, 10:23 AM

I'm fairly sure 10k requests will fit in the included usage

Ssathoro using DO as a queue should work well. you would need to do the pricing calculati...

A

AdvanyOP•5/31/23, 10:58 AM

I just saw that queues require workers to be on the unbound plan. So that means costs of using queue + workers or DO's would be the same

AAdvany I just saw that queues require workers to be on the unbound plan. So that means ...

S

sathoro•5/31/23, 10:59 AM

I suggested DOs based on developer experience. we are talking about pennies anyways so I wasn't too worried about cost

A

AdvanyOP•5/31/23, 11:00 AM

yeah, the 10k was only a sample. It will be more like 50 million in production

AAdvany yeah, the 10k was only a sample. It will be more like 50 million in production

S

sathoro•5/31/23, 11:01 AM

yeah then I would definitely use DOs. Queues is a beta product that has low concurrency and additional latency. with DOs you could create unlimited queues for extra concurrency

S

sathoro•5/31/23, 11:02 AM

this is what I do in prod. a DO for each user as a queue. works great

Ssathoro yeah then I would definitely use DOs. Queues is a beta product that has low conc...

A

AdvanyOP•5/31/23, 11:02 AM

yeah. I might even go for a non serverless solution. Seems serverless is really expensive for this use case.

AAdvany yeah. I might even go for a non serverless solution. Seems serverless is really ...

S

sathoro•5/31/23, 11:02 AM

I was thinking about suggesting that since you seemed to be hung up on cost and this is something where a $5 VPS would be the most cost effective thing. just depends on your needs

Ssathoro I was thinking about suggesting that since you seemed to be hung up on cost and ...

A

AdvanyOP•5/31/23, 11:09 AM

yeah. the client needs a cheaper solution in this case... my calculations seem to amount to around $600 for 25 million requests queued and handled

DO seconds 25m * 15 sec = 375 million seconds
DO price = 375 million seconds * 128mb/1gb * $12.50 = $603

+ storage costs + request costs

I think a $100 vps might do a better job handling the use case, unless my calculations are wrong

AAdvany yeah. the client needs a cheaper solution in this case... my calculations seem t...

S

sathoro•5/31/23, 11:11 AM

that sounds like a lot but that is 4,340 days in processing time lol. that is a heck of a lot of requests especially when they take so long...

A

AdvanyOP•5/31/23, 11:12 AM

yeah. It is... need it to prerender sitemaps... and chrome isn't that fast lol

AAdvany yeah. It is... need it to prerender sitemaps... and chrome isn't that fast lol

S

sathoro•5/31/23, 11:13 AM

ah...

Ssirupsen At what point of monthly bandwidth to the cache API does someone from Cloudflare...

H

Hello, I’m Allie!•5/31/23, 11:29 AM

Yeah, CF doesn’t provide an explicit public limit for various reasons. If you are a non-profit/OSS provider, then you might qualify for a discount. Pricing is something you need to contact sales for

H

Hello, I’m Allie!•5/31/23, 11:29 AM

For anything above PAYGO, it won’t be published

Ssathoro ah...

A

AdvanyOP•5/31/23, 11:57 AM

maybe I can do 4 requests at a time inside a do or worker customer with parallel primoses... didnt think of that. That should lower the cost to $150 a month...

AAdvany maybe I can do 4 requests at a time inside a do or worker customer with parallel...

S

sathoro•5/31/23, 12:09 PM

yeah you can do like 8 at a time I think

S

sathoro•5/31/23, 12:28 PM

that reminds me, is it normal not to get any replies on limit increase requests? I've done several spanning several weeks and never heard anything

S

sathoro•5/31/23, 12:28 PM

(I'm not spamming them, they are for different limits)

Ssirupsen 👍 I'll try to get on a call with someone from sales

K

kian•5/31/23, 12:37 PM

@rickyrobinett might be able to get you in touch with the right people. I suspect it'll differ based on if you're just using the CDN (i.e not using any other Cloudflare product) or if you're paying for Workers/R2/etc as what's generating that cache(able) content but they can confirm.

R

rickyrobinett•5/31/23, 12:40 PM

Yes, @sirupsen. Happy to help navigate. I’ll DM you.

S

sathoro•5/31/23, 12:51 PM

is the built-in fetch caching supposed to work locally? I'm doing this and it doesn't seem to be caching it (with

--local

--local

--local

--local)

const resp = await fetch(`...`, {
  cf: {
    cacheTtl: 60 * 10,
  },
});

const resp = await fetch(`...`, {
  cf: {
    cacheTtl: 60 * 10,
  },
});

const resp = await fetch(`...`, {
  cf: {
    cacheTtl: 60 * 10,
  },
});

const resp = await fetch(`...`, {
  cf: {
    cacheTtl: 60 * 10,
  },
});

S

sathoro•5/31/23, 1:04 PM

just tested on a real worker and it isn't working either, what am I missing? Hmm

KKartik How many emails can i send using email workers?🧐

S

sathoro•5/31/23, 1:10 PM

aren't those for receiving emails? if you are sending emails you will need to use a third-party service like Postmark (unless you don't care about them all going to spam) and your limits would be between you and them

KKartik I think we can send emails also🤫

D

DaniFoldi•5/31/23, 1:17 PM

you can only send to verified addresses, which should really be just you/team members, to give you notifications of eg. critical errors, or whatever you set it up as

D

DaniFoldi•5/31/23, 1:17 PM

a good example I have is a contact form, you can receive an email to the address you verified, and you can then reply if you'd like to

KKartik Umm, but that depends on the address i have provided in the binding, but if i do...

D

DaniFoldi•5/31/23, 1:21 PM

you can then send to all verified addresses of the account

L

lokiwind•5/31/23, 1:58 PM

catch (error) {
          console.error(`Error while inserting row into accesslog table: ${error}`);
          return Response.json({ message: `Error while inserting row into accesslog table: ${error.message}` });
        }

catch (error) {
          console.error(`Error while inserting row into accesslog table: ${error}`);
          return Response.json({ message: `Error while inserting row into accesslog table: ${error.message}` });
        }

catch (error) {
          console.error(`Error while inserting row into accesslog table: ${error}`);
          return Response.json({ message: `Error while inserting row into accesslog table: ${error.message}` });
        }

catch (error) {
          console.error(`Error while inserting row into accesslog table: ${error}`);
          return Response.json({ message: `Error while inserting row into accesslog table: ${error.message}` });
        }

'error' of type 'unknown'.

L

lokiwind•5/31/23, 1:59 PM

What am I doing wrong?

Llokiwind What am I doing wrong?

H

Hello, I’m Allie!•5/31/23, 2:00 PM

Try

try {
  // ...
} catch (error: any) {
  console.error(`Error while inserting row into accesslog table: ${error}`);
  return Response.json({ message: `Error while inserting row into accesslog table: ${error.message}` });
}

try {
  // ...
} catch (error: any) {
  console.error(`Error while inserting row into accesslog table: ${error}`);
  return Response.json({ message: `Error while inserting row into accesslog table: ${error.message}` });
}

try {
  // ...
} catch (error: any) {
  console.error(`Error while inserting row into accesslog table: ${error}`);
  return Response.json({ message: `Error while inserting row into accesslog table: ${error.message}` });
}

try {
  // ...
} catch (error: any) {
  console.error(`Error while inserting row into accesslog table: ${error}`);
  return Response.json({ message: `Error while inserting row into accesslog table: ${error.message}` });
}

L

lokiwind•5/31/23, 2:01 PM

try {
          await env.DB.prepare(
            "INSERT INTO accesslog (ip, time, path) VALUES (?1, ?2, ?3)"
          ).bind(ip, time, pathname).run();
        } catch (error) {
          console.error(`Error while inserting row into accesslog table: ${error}`);
          return Response.json({ message: `Error while inserting row into accesslog table: ${error.message}` });
        }
      }

try {
          await env.DB.prepare(
            "INSERT INTO accesslog (ip, time, path) VALUES (?1, ?2, ?3)"
          ).bind(ip, time, pathname).run();
        } catch (error) {
          console.error(`Error while inserting row into accesslog table: ${error}`);
          return Response.json({ message: `Error while inserting row into accesslog table: ${error.message}` });
        }
      }

try {
          await env.DB.prepare(
            "INSERT INTO accesslog (ip, time, path) VALUES (?1, ?2, ?3)"
          ).bind(ip, time, pathname).run();
        } catch (error) {
          console.error(`Error while inserting row into accesslog table: ${error}`);
          return Response.json({ message: `Error while inserting row into accesslog table: ${error.message}` });
        }
      }

try {
          await env.DB.prepare(
            "INSERT INTO accesslog (ip, time, path) VALUES (?1, ?2, ?3)"
          ).bind(ip, time, pathname).run();
        } catch (error) {
          console.error(`Error while inserting row into accesslog table: ${error}`);
          return Response.json({ message: `Error while inserting row into accesslog table: ${error.message}` });
        }
      }

@HardAtWork

Llokiwind ```js try { await env.DB.prepare( "INSERT INTO accesslog (...

H

Hello, I’m Allie!•5/31/23, 2:01 PM

Yeah, so add the

any

any

any

any type, and see if it works.

Llokiwind ```js try { await env.DB.prepare( "INSERT INTO accesslog (...

L

lokiwind•5/31/23, 2:02 PM

The code is now like this

L

lokiwind•5/31/23, 2:02 PM

But error :'error' of type 'unknown'.

Llokiwind The code is now like this

H

Hello, I’m Allie!•5/31/23, 2:03 PM

Yeah, so add the

any

any

any

any type to the catch block like so, and see if it stops throwing that error.

try {
    await env.DB.prepare(
      "INSERT INTO accesslog (ip, time, path) VALUES (?1, ?2, ?3)"
    ).bind(ip, time, pathname).run();
  } catch (error: any) {
    console.error(`Error while inserting row into accesslog table: ${error}`);
    return Response.json({ message: `Error while inserting row into accesslog table: ${error.message}` });
  }
}

try {
    await env.DB.prepare(
      "INSERT INTO accesslog (ip, time, path) VALUES (?1, ?2, ?3)"
    ).bind(ip, time, pathname).run();
  } catch (error: any) {
    console.error(`Error while inserting row into accesslog table: ${error}`);
    return Response.json({ message: `Error while inserting row into accesslog table: ${error.message}` });
  }
}

try {
    await env.DB.prepare(
      "INSERT INTO accesslog (ip, time, path) VALUES (?1, ?2, ?3)"
    ).bind(ip, time, pathname).run();
  } catch (error: any) {
    console.error(`Error while inserting row into accesslog table: ${error}`);
    return Response.json({ message: `Error while inserting row into accesslog table: ${error.message}` });
  }
}

try {
    await env.DB.prepare(
      "INSERT INTO accesslog (ip, time, path) VALUES (?1, ?2, ?3)"
    ).bind(ip, time, pathname).run();
  } catch (error: any) {
    console.error(`Error while inserting row into accesslog table: ${error}`);
    return Response.json({ message: `Error while inserting row into accesslog table: ${error.message}` });
  }
}

L

lokiwind•5/31/23, 2:04 PM

Thank you I will try

L

lokiwind•5/31/23, 2:07 PM

Problem solved thanks @HardAtWork

M

Murder Chicken•5/31/23, 4:04 PM

Random question. Has anyone found an HTML minifier that works in a CF worker? Most of them fail with missing dependencies like:

  node_modules/clean-css/lib/options/format.js:1:30: ERROR: Could not resolve "os"
  node_modules/clean-css/lib/options/inline-request.js:1:18: ERROR: Could not resolve "url"
  node_modules/clean-css/lib/options/rebase-to.js:1:19: ERROR: Could not resolve "path"
  node_modules/clean-css/lib/reader/apply-source-maps.js:1:17: ERROR: Could not resolve "fs"
  node_modules/clean-css/lib/reader/apply-source-maps.js:2:19: ERROR: Could not resolve "path"

  node_modules/clean-css/lib/options/format.js:1:30: ERROR: Could not resolve "os"
  node_modules/clean-css/lib/options/inline-request.js:1:18: ERROR: Could not resolve "url"
  node_modules/clean-css/lib/options/rebase-to.js:1:19: ERROR: Could not resolve "path"
  node_modules/clean-css/lib/reader/apply-source-maps.js:1:17: ERROR: Could not resolve "fs"
  node_modules/clean-css/lib/reader/apply-source-maps.js:2:19: ERROR: Could not resolve "path"

  node_modules/clean-css/lib/options/format.js:1:30: ERROR: Could not resolve "os"
  node_modules/clean-css/lib/options/inline-request.js:1:18: ERROR: Could not resolve "url"
  node_modules/clean-css/lib/options/rebase-to.js:1:19: ERROR: Could not resolve "path"
  node_modules/clean-css/lib/reader/apply-source-maps.js:1:17: ERROR: Could not resolve "fs"
  node_modules/clean-css/lib/reader/apply-source-maps.js:2:19: ERROR: Could not resolve "path"

  node_modules/clean-css/lib/options/format.js:1:30: ERROR: Could not resolve "os"
  node_modules/clean-css/lib/options/inline-request.js:1:18: ERROR: Could not resolve "url"
  node_modules/clean-css/lib/options/rebase-to.js:1:19: ERROR: Could not resolve "path"
  node_modules/clean-css/lib/reader/apply-source-maps.js:1:17: ERROR: Could not resolve "fs"
  node_modules/clean-css/lib/reader/apply-source-maps.js:2:19: ERROR: Could not resolve "path"

M

Murder Chicken•5/31/23, 4:05 PM

All of the ones I've looked at have similar dependencies.

D

dave•5/31/23, 4:35 PM

Is there a way to match on every path except for one? https://developers.cloudflare.com/pages/platform/headers/

Headers · Cloudflare Pages docs

To attach headers to Cloudflare Pages responses, create a _headers plain text file in the output folder of your project. It is usually the folder that …

D

dave•5/31/23, 4:36 PM

oh I guess I can detach the header for only one path

I need to do 10k fetches to api end point. Responses can take up to 15 seconds. What would be the co

Similar Threads

Similar Threads

Similar Threads