Hi, could please someone have a look at this thread and tell me is queues could be a culprit here? h

Hi, could please someone have a look at this thread and tell me is queues could be a culprit here? kv

Wwarpandas Is there any reason why I can't get the Authorization part of the request sent t...

Can you expand? There’s nothing different about a Worker that calls .send() on a queue that would inhibit any Request headers.

Kkika Hi, could please someone have a look at this thread and tell me is queues could ...

Matt Silverlock•3/17/23, 2:03 PM

Can you:

1. Paste the exact error message you're seeing? Is it resources exceeded or is it something else?
2. Can you confirm your batch settings, as specified in your

wrangler.toml

wrangler.toml

- ?
3. Share your accountId with me?

MMatt Silverlock Can you: 1. Paste the exact error message you're seeing? Is it resources exceed...

kikaOP•3/17/23, 2:27 PM

I'm on paid, unbound plan, so 600ms should not be my CPU limit.

kian•3/17/23, 2:32 PM

Exceeds Resource could also show if you run out of memory, which is 128MB

Kkian Exceeds Resource could also show if you run out of memory, which is 128MB

kikaOP•3/17/23, 2:32 PM

at exactly 1000 items? never 999, never 1001

Kkika at exactly 1000 items? never 999, never 1001

kikaOP•3/17/23, 2:35 PM

also, it's highly unlikely. a few hundred K file, HTML with inserted badly formed JSON. it's messy, but not large and not complex. I believe I measured the parser footprint early in the development, it was a couple M if I recall correctly.

kian•3/17/23, 2:53 PM

The only limit that’s specifically 1,000 is the amount of in-house subrequests per invocation, which afaik should just be a rejected promise on the subrequest rather than a resources exceeded warning. That warning should exclusively be CPU or memory.

Kkian The only limit that’s specifically 1,000 is the amount of in-house subrequests p...

kikaOP•3/17/23, 2:57 PM

maybe my (or Rust's worker runtime's) error handling is leaking memory. dunno how to test for that, needless to say that the worker works okay locally. strange though, this code is WIP and littered with

unwrap()

unwrap()

s which turn every error into a

panic!()

panic!()

kian•3/17/23, 3:08 PM

Do you see it in the headers log?

kian•3/17/23, 3:18 PM

How are you making the request to the Worker? It doesn’t sound like that header is getting there at all

kian•3/17/23, 3:20 PM

Lots of your fetch options seem un-necessary - but I’m pretty sure no-cors doesn’t allow you to send that header.

Wwarpandas ``` Access to fetch at 'https:// test-worker.sensay.workers.dev/' from origin 'h...

Chaika•3/18/23, 7:26 PM

It sounds like you are trying to fetch your Worker Script from your browser to add to a queue, on a webpage at http://localhost:8080

The point of CORS is to prevent other websites/origins from fetching your origin, preventing scripts from accessing resources that they shouldn't have access to.

"no-cors" isn't a magical escape hatch to get around this, it's a very restrictive mode to make your request safe. Stripping most headers from the request, restricting content-type, restricting http request type, and making you unable to read the response body/status/headers: https://evertpot.com/no-cors/

The answer is to change your worker to respond with the required CORS Headers, for example:
Access-Control-Allow-Origin: http://localhost:8080
Your real website won't be on localhost:8080, for local testing you could use a proxy that adds the CORS headers for you. For production use though, if you want to continue to have your script on a different website/origin then your website, you will need to add the required CORS Headers.
More info: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
This has nothing to do with Queues or even Workers themselves, just do to with Cross-origin requests, so if you have any more questions/etc, please use #coding-help or another channel that fits better

Matt Silverlock•3/20/23, 11:48 PM

Queue consumer concurrency is going to land in just under two weeks: https://developers.cloudflare.com/queues/changelog/#consumer-concurrency-upcoming

If you have a use-case where you want to limit concurrency, you’ll want to cap it via the dashboard or wrangler before we enable autoscaling.

Changelog · Cloudflare Queues

Queue consumers will soon automatically scale up concurrently as a queues’ backlog grows in order to keep overall message processing latency down. …

MMatt Silverlock Queue consumer concurrency is going to land in just under two weeks: https://dev...

Erisa•3/21/23, 8:18 AM

@Queues

zegevlier•3/21/23, 1:06 PM

Yup, that's totally fine

MMoretti Thinking about this use case: fetch handler receives request, process and needs...

Matt Silverlock•3/21/23, 4:30 PM

Are you trying to serve a real-time response back to a user? If so, how do you intend to do that?

Matt Silverlock•3/21/23, 4:31 PM

Queues is best at async work. If you're relying on returning a live response to a client, then it may not make sense, but you'll also need to manage your upstream API (in this case, OpenAI's) rate limits as well vs. your own user demand.

Kai•3/21/23, 5:27 PM

doing something like fetch -> upgrade WS -> push WS to DO & enqueue work item w/ DO Id -> ... queue ... -> eventual return via DO WS -> close WS works decently imo. Can reuse the WS / DO as needed & is decently efficient. It'll be a bit more expensive depending on how you implement it (DOs with WS are never evicted so you pay) but it's possible to do & even implement timeouts and that sort of thing.

MMoretti In our case I want to close the connection with the user and then reply async wi...

Matt Silverlock•3/21/23, 6:44 PM

How do you get that response back to the user? e.g. what medium?

MMatt Silverlock How do you get that response back to the user? e.g. what medium?

Matt Silverlock•3/21/23, 6:44 PM

(does the user/client poll/retry?)

MMoretti I opened a support ticket yesterday. I can't enable queues on my enterprise acco...

Matt Silverlock•3/22/23, 7:56 PM

Can you DM me your account ID / reach out to your account team? Why can’t you enable it? (what’s the error?)

MMoretti I opened a support ticket yesterday. I can't enable queues on my enterprise acco...

Matt Silverlock•3/22/23, 7:56 PM

Also: queues

Unsmart•3/22/23, 10:39 PM

What happens when you press agree?

Matt Silverlock•3/22/23, 11:01 PM

(We addressed this via DMs. They’re on a legacy subscription with ~14 accounts. Addressing it directly)

KKlowner ah ha, limited to one active consumer!

Emilio•3/23/23, 6:38 PM

Bingo! Our Queues can only have 1 consumer at a time.

Jacob•3/24/23, 2:43 AM

When using ack() on each messages, under what conditions will it retry the non ack’d messages? Is it only if the worker crashes or retry() is called?

Jacob•3/24/23, 2:58 AM

From reading the change log, it looks like non ack’d messages will only be retried when the consumer crashes (throws an unhandled error.)
I’ll add some retry()’s to my consumer that try/catches everything

Hi, could please someone have a look at this thread and tell me is queues could be a culprit here? h

Similar Threads

Similar Threads

Similar Threads