I don't think I hit any limit, because this error usually is the first request of the day / first re

I don't think I hit any limit, because this error usually is the first request of the day / first request after deploy new version to CF worker, and the status 500 is coming from mailchannels api

SSheng I don't think I hit any limit, because this error usually is the first request o...

johtso•5/10/24, 12:28 PM

sorry, that wasn't in response to you, was an unrelated question

Jjohtso what happens when you hit the subrequest limit, does fetch throw an error?

kian•5/10/24, 12:40 PM

Yes

kian•5/10/24, 12:41 PM

So subrequests work in two ways - there are in-house subrequests, and everything else.

kian•5/10/24, 12:41 PM

They have separate counters, separate 'buckets'

kian•5/10/24, 12:41 PM

You can (on a paid plan) make 1000 subrequests with something like fetch(url)fetch(url), and another 1000 to in-house things (KV, R2, DO, D1, you get the idea)

kian•5/10/24, 12:41 PM

Once you hit that limit, they throw a "subrequest limit exceeded" exception

johtso•5/10/24, 12:55 PM

@kian thanks for clarifying!

johtso•5/10/24, 12:56 PM

still not entirely sure how to deal with these limits when I'm doing work that requires making a lot of requests

johtso•5/10/24, 1:10 PM

My instinct is to catch that error at a high level, and then schedule an immediate alarm

johtso•5/10/24, 1:11 PM

Scheduling an alarm doesn't count as a request does it?

johtso•5/10/24, 1:13 PM

Ah yes, they're transactional storage API operations, that's good

johtso•5/10/24, 1:21 PM

@kian is there anywhere I can see exactly how to catch that exception?

kian•5/10/24, 1:23 PM

Just a normal

try/catch

try/catch

around the subrequest

Kkian Just a normal `try/catch` around the subrequest

johtso•5/10/24, 1:54 PM

but how to I check if an error is a "subrequest limit exceeded" exception? is that the exact name of the exception? e.name === "subrequest limit exceeded"e.name === "subrequest limit exceeded"?

SSheng Mailchannels + CF Workers, anyone facing status 500 error, EOF when sending emai...

James•5/10/24, 1:57 PM

Sadly not a direct solution to your problem, but I would recommend you avoid MailChannels. Cloudflare's continued partnership with them is very disappointing after their security issues and terrible response. See https://canary.discord.com/channels/595317990191398933/779390076219686943/1226529017763463219 for more details.

Jjohtso but how to I check if an error is a "subrequest limit exceeded" exception? is th...

kian•5/10/24, 1:58 PM

export default {
  async fetch() {
    try {
      while (true) { await fetch("https://cloudflare.com/cdn-cgi/trace") }
    } catch (e) {
      return new Response(JSON.stringify(e, Object.getOwnPropertyNames(e)))
    }
  },
};

export default {
  async fetch() {
    try {
      while (true) { await fetch("https://cloudflare.com/cdn-cgi/trace") }
    } catch (e) {
      return new Response(JSON.stringify(e, Object.getOwnPropertyNames(e)))
    }
  },
};

kian•5/10/24, 1:58 PM

You can test it with that and see what the error is

Kkian ```js export default { async fetch() { try { while (true) { await fe...

johtso•5/10/24, 1:59 PM

I'm guessing it's along the lines of this? https://github.com/cloudflare/miniflare/blob/c2ed3afdc1fed9f78d5cb6c50edc793a9a43a850/packages/shared/src/context.ts#L149

johtso•5/10/24, 2:00 PM

which doesn't seem very amenable to being handled specifically

johtso•5/10/24, 2:01 PM

Is this just the javascript way? generic errors with error text?

johtso•5/10/24, 2:01 PM

coming from python I'm used to specific subclasses of errors

James•5/10/24, 2:02 PM

You absolutely can create subclasses of errors in JS, and attach things like a

code

code

, etc. but not everything does, sadly

johtso•5/10/24, 2:03 PM

would be nice if core cloudflare errors did this :/

johtso•5/10/24, 2:10 PM

{"stack":"Error: Too many subrequests.\n    at Object.fetch (worker.js:15:28)","message":"Too many subrequests."}

{"stack":"Error: Too many subrequests.\n    at Object.fetch (worker.js:15:28)","message":"Too many subrequests."}

johtso•5/10/24, 2:19 PM

and this is going to behave differently when ran locally?

johtso•5/10/24, 2:20 PM

@kian and do you think I can safely run that snippet locally, or will I end up spamming cloudflare?

johtso•5/10/24, 2:24 PM

I don't think local wrangler is respecting subrequest limits..

johtso•5/10/24, 2:51 PM

should I be able to test locally with the same subrequest limit behaviour? the github issues I'm seeing give the impression that it should "just work"

James•5/10/24, 5:34 PM

I don't believe subrequest limits are enforced at all locally

James•5/10/24, 5:35 PM

so that'll probably just run forever

johtso•5/10/24, 6:29 PM

@James that seems unfortunate? something can seem to run fine locally and then break when deployed

James•5/10/24, 6:29 PM

I agree, it's very much not ideal

James•5/10/24, 6:29 PM

https://github.com/cloudflare/workers-sdk/issues/4359 seems to be where this is tracked if you want to add your thoughts

johtso•5/10/24, 6:30 PM

https://github.com/cloudflare/workerd/blob/a3de83cbbd338bb64f9782f3584ec17ad6995821/src/workerd/io/limit-enforcer.h#L84

johtso•5/10/24, 6:30 PM

is this not being used when running locally?

johtso•5/10/24, 6:30 PM

oh right, workerd is the production runtime, but also used locally

Walshy•5/10/24, 6:31 PM

that is local but the implementation is internal, that's just the header file

Walshy•5/10/24, 6:34 PM

how come you're doing 1k subrequests?

codebam•5/10/24, 6:53 PM

did cloudflare AI just go down? I didn't change any code and I'm getting Error: AiError: undefined: ERROR 3040: Unknown internal errorError: AiError: undefined: ERROR 3040: Unknown internal error

Walshy•5/10/24, 6:54 PM

https://canary.discord.com/channels/595317990191398933/895794943182909470/1238495826683101194

WWalshy https://canary.discord.com/channels/595317990191398933/895794943182909470/123849...

codebam•5/10/24, 6:55 PM

thanks

WWalshy how come you're doing 1k subrequests?

johtso•5/10/24, 7:17 PM

I'm mirroring data to my d1 database from an api that only allows requesting one thing at a time

Walshy•5/10/24, 7:18 PM

i'd submit to a queue and process in a queue

WWalshy i'd submit to a queue and process in a queue

johtso•5/10/24, 7:25 PM

just seems a bit overkill.. I don't even want parallelism because I don't want to overload my API

johtso•5/10/24, 7:26 PM

and I don't really need a concept of a "job" or queue item because anything not already in my database is something that still needs to be fetched

johtso•5/10/24, 7:26 PM

alternatively I was thinking I might just run the "backfilling" as a local process on my computer

johtso•5/10/24, 7:27 PM

as it's quite different from the work of constantly monitoring for new data (which should only involve a few requests)

liam | helius.dev•5/10/24, 10:32 PM

Is anyone using workers to proxy websockets getting 502 errors currently? Suddenly all our websockets, across multiple accounts, stopped working

Lliam | helius.dev Is anyone using workers to proxy websockets getting 502 errors currently? Sudden...

Walshy•5/10/24, 10:53 PM

Hey, got a repro URL? and is this fully client <-> Worker or is there an origin involved?

WWalshy Hey, got a repro URL? and is this fully client <-> Worker or is there an origin ...

liam | helius.dev•5/10/24, 10:57 PM

Origin, normally connects to a cloudflare LB, but I have now tried arbitrary origins that support wss and it all returns 502. It also swallows our logs so its like something is broken in the upgrade path

You can try making a wss connection to here `atlas-mainnet.helius-rpc.com?

I don't think I hit any limit, because this error usually is the first request of the day / first re

Similar Threads

Similar Threads

Similar Threads