Any chance you can share how you did this? I've still not figured out an efficient way to do somethi

Any chance you can share how you did this? I've still not figured out an efficient way to do something like this.

KaiOP•3/14/23, 10:20 PM

Also, unrelated, is there a chance we can get a sort of dynamic key as part of queues? There's still only one queue and one consumer script but batching & concurrency is done per-key?
Seems like an easy way to enable much more reuse. Creating a new queue for every log file that needs appending or for every API region etc. is annoying and feels unnecessary, they are all exact duplicates just with a different name to make them work in parallel

KKai Any chance you can share how you did this? I've still not figured out an efficie...

Jacob•3/14/23, 10:35 PM

It’s not terribly clever… I just watch for it to return headers saying it’s out of quota and await scheduler.wait() with whatever duration it gives in another header. Also sleeps if I get back a 429 (based on the retry_after value it sends.) I do a bit of extra sleeping after like 4x 429s in a short period to be nice, but it’s not entirely necessary.
Discord has good documentation for the rate limit headers and 429 responses it sends: https://discord.com/developers/docs/topics/rate-limits

Discord Developer Portal

Discord Developer Portal — API Docs for Bots and Developers

Integrate your service with Discord — whether it's a bot or a game or whatever your wildest imagination can come up with.

KaiOP•3/14/23, 10:36 PM

await scheduler.wait()

this is probably the bit I'm missing? what is that? Does that sleep but isn't billable / cause timeouts?

Jacob•3/14/23, 10:38 PM

Basically the same as the classic await new Promise(r => setTimeout(r, 2000)); AFAIK. You will get billed for that duration

Jacob•3/14/23, 10:38 PM

Discord only ever makes me wait like 2 seconds so I haven’t run into timeouts

KaiOP•3/14/23, 10:39 PM

ah ok :/ I have much tighter & complex rate limits and have so far failed to implement them on top of cloudflare without also paying a lot. Like 10k+ outstanding requests but only 200 / 1h allowed & any overrun disqualifies me from getting higher limits so

Jacob•3/14/23, 10:41 PM

Ahh yeah if it’s that tight then Queues might not work for you

KaiOP•3/14/23, 10:41 PM

yeah, but thanks for sharing anyways!

Jacob•3/14/23, 10:43 PM

Maybe you could specify the file it should append to in the Queue message? Each message you send is an object, so you could send something like {file: “log1.txt”, line: “my log line 123”}

Jacob•3/14/23, 10:43 PM

Then the consumer would check that field and append to that file

KaiOP•3/14/23, 10:43 PM

yeah but then the batching gets worse and worse the more files I funnel through that one

Jacob•3/14/23, 10:44 PM

Consumer concurrency is coming soon

KaiOP•3/14/23, 10:44 PM

Yeah, excited for that feature, but I don't see how that interacts with this

Jacob•3/14/23, 10:44 PM

Ohh sorry just understood what you meant

Jacob•3/14/23, 10:45 PM

I see so a batch has a mix of messages from all the different files, so you have more writes to your log storage

Jacob•3/14/23, 10:45 PM

Interesting use case. I wonder if there will be some kind of partitioning in the future

KaiOP•3/14/23, 10:45 PM

yeah and I have a bunch of essentially duplicated logic then + either have to use huge batch sizes and risk overrunning the time, or have smaller sizes, which may end up in unreasonably small write batches

KaiOP•3/14/23, 10:46 PM

I mean duplicating the queues has worked for me in the past but it makes the calling code more confusing then necessary, makes the dashboard worse and updating is annoying

KaiOP•3/14/23, 10:47 PM

I mean all this is solved on my side fairly easily with just a bit of TS magic & a few scripts but seems like a fairly easily thing to solve on the queues side

KaiOP•3/14/23, 10:47 PM

(even if I have to set the possible keys ahead of time & can't have too many, it'd still be so nice)

KaiOP•3/14/23, 10:49 PM

unfortunately I have no insight into what other people use queues for, but batching writes & rate limiting are the two big ones I've seen so far, and both would benefit from something there greatly

Unsmart•3/14/23, 10:54 PM

I don't think queues would be any good for rate limiting unless you mean reducing throughput to external apis

UUnsmart I don't think queues would be any good for rate limiting unless you mean reducin...

Jacob•3/14/23, 10:57 PM

That’s exactly what I’m doing

UUnsmart I don't think queues would be any good for rate limiting unless you mean reducin...

KaiOP•3/14/23, 11:54 PM

yeah that's also what I'm doing. I think (?) a fairly common need given cloudflare based products have no problem handing out request volumes that just kill vendors. Generally I've not seen that many external APIs disallow hitting the rate limit in general, but they'll get unhappy if it's too much / impacting other customers.

KaiOP•3/14/23, 11:55 PM

So at least taking some care to stop that from happening is important and queues seems like the best way to still ensure all requests make it eventually, but limit throughput. It's just a bit akward to implement efficiently right now

Erisa•3/14/23, 11:58 PM

you can also batch from a queue if the provider allows it, to avoid spending a request for each and better optimise the amount of data you send in each request (which i think jacob is doing?)

Jacob•3/15/23, 12:39 AM

Yes that’s what I’m doing to increase throughout - batching together data so I’m making fewer api calls

Jacob•3/15/23, 12:41 AM

One of my queues is saving batches of data to R2 which saves a ton of API calls (averages around 30 messages/write to R2)

Erisa•3/15/23, 12:42 AM

Thats awesome, reminds me of how logpush will batch requests in a similar way

Jacob•3/15/23, 12:46 AM

Nice! Yeah it’s super nice. I also have a Workers cron that will combine each hour of files into a single file so that it’s super easy to process later

$\ ឵឵឵$ \\ ឵឵឵Hey all, is it the plan to raise the retention limits from 4 days in future?

Matt Silverlock•3/16/23, 5:23 PM

Yes, once we start actively expiring messages that reach the retention limit, and build some controls around it. One of the nearer-term items on the roadmap.

$\ ឵឵឵$ \\ ឵឵឵Awesome. Do you guys have any timeline in mind for the delay and redrive feature...

Matt Silverlock•3/16/23, 6:31 PM

No timeline just yet. HTTP Pull is the big next thing on the roadmap, so that folks can consume from outside Workers and/or in cases where you have a pull-based workload (vs. pushing, as we do today)

kika•3/16/23, 9:59 PM

Hi, could please someone have a look at this thread and tell me is queues could be a culprit here? kv

Wwarpandas Is there any reason why I can't get the Authorization part of the request sent t...

Matt Silverlock•3/17/23, 1:35 PM

Can you expand? There’s nothing different about a Worker that calls .send() on a queue that would inhibit any Request headers.

Kkika Hi, could please someone have a look at this thread and tell me is queues could ...

Matt Silverlock•3/17/23, 2:03 PM

Can you:

1. Paste the exact error message you're seeing? Is it resources exceeded or is it something else?
2. Can you confirm your batch settings, as specified in your

wrangler.toml

wrangler.toml

- ?
3. Share your accountId with me?

MMatt Silverlock Can you: 1. Paste the exact error message you're seeing? Is it resources exceed...

kika•3/17/23, 2:27 PM

I'm on paid, unbound plan, so 600ms should not be my CPU limit.

kian•3/17/23, 2:32 PM

Exceeds Resource could also show if you run out of memory, which is 128MB

Kkian Exceeds Resource could also show if you run out of memory, which is 128MB

kika•3/17/23, 2:32 PM

at exactly 1000 items? never 999, never 1001

Kkika at exactly 1000 items? never 999, never 1001

kika•3/17/23, 2:35 PM

also, it's highly unlikely. a few hundred K file, HTML with inserted badly formed JSON. it's messy, but not large and not complex. I believe I measured the parser footprint early in the development, it was a couple M if I recall correctly.

kian•3/17/23, 2:53 PM

The only limit that’s specifically 1,000 is the amount of in-house subrequests per invocation, which afaik should just be a rejected promise on the subrequest rather than a resources exceeded warning. That warning should exclusively be CPU or memory.

Kkian The only limit that’s specifically 1,000 is the amount of in-house subrequests p...

kika•3/17/23, 2:57 PM

maybe my (or Rust's worker runtime's) error handling is leaking memory. dunno how to test for that, needless to say that the worker works okay locally. strange though, this code is WIP and littered with unwrap()unwrap() s which turn every error into a panic!()panic!()

kian•3/17/23, 3:08 PM

Do you see it in the headers log?

kian•3/17/23, 3:18 PM

How are you making the request to the Worker? It doesn’t sound like that header is getting there at all

Any chance you can share how you did this? I've still not figured out an efficient way to do somethi

Similar Threads

Similar Threads

Similar Threads