Hey @Matt , how's the investigation going, do you have any updates on timelines? Should I just assum

Hey @Matt , how's the investigation going, do you have any updates on timelines? Should I just assume that the fix would be in the GA in Q1/Q2 next year?

Ccg Hey @Matt , how's the investigation going, do you have any updates on timelines?...

Matt Silverlock•12/11/23, 12:39 AM

We’re still working on it. Non-trivial to fix. Likely within the next 2-3 weeks.

SSam We have also had this same issue with a queue that seemly just stopped sending m...

Matt Silverlock•12/11/23, 7:45 PM

We’re working on addressing this - it’s an unfortunate beta-type bug.

SSam We have also had this same issue with a queue that seemly just stopped sending m...

Matt Silverlock•12/11/23, 7:46 PM

P.S: an account ID and queue ID allow us to investigate further. I generally recommend avoiding assuming your bug == someone else’s bug

Matt Silverlock•12/11/23, 7:48 PM

They’re not private identifiers. You can post them here @SamP

SSam account ID is 12fd8847ed86e95913678748fab30104 and broken queue ID is seine

Matt Silverlock•12/11/23, 8:57 PM

Looks like the consumer is consuming fast now. We manually intervened (and have a patch going out to attempt to fix the root cause)

Ccodehex @Matt We have this same issue with queue. Account ID: d2025e363dd2037ea27e179562...

Matt Silverlock•12/12/23, 3:51 AM

Can you describe the issue? How large is your backlog?

tdevthairun•12/12/23, 4:37 AM

hello I need help about cloudflare queue I want to send queue to run hono routes for create data in postgresql or any idea for help me.

tdevthairun•12/12/23, 4:37 AM

VVâjhâtz What would be the best way to process a million entries? I'm currently testing d...

Matt Silverlock•12/18/23, 1:25 PM

Stopped locally? Do you see errors? Can you open a bug report on GitHub (including the wrangler version & your code) - ?

thiccvitalik•12/21/23, 4:05 PM

Hi! When I have a consumer with a max_batch_size of 5, does this imply that the processing of those 5 messages will share the 128MB memory allocated to the worker? And if I set the max_batch_size to 1, does it ensure that the processing of each message will exclusively have 128MB of memory?

Tthiccvitalik Hi! When I have a consumer with a max_batch_size of 5, does this imply that the ...

Unsmart•12/21/23, 9:42 PM

If you have a batch of 5 all messages will be in the same invocation which has 128MB. But also in addition any concurrency you set on the queue would also share that same 128MB afaik. It might be possible that even http/cron events that happen to hit the same server share the 128MB if on the same worker (although not too sure about that).

From what I understand basically a worker script will have 128MB shared across the entire server not just the single invocation.

thiccvitalik•12/21/23, 9:54 PM

Thanks for the answers. That's very surprising that additional concurrency also shares the 128MB. The maximum concurrent consumer invocations is 20 in beta. That would mean each of those has only 6MB. blob_sweat

Unsmart•12/21/23, 9:59 PM

Im not entirely sure if thats accurate or if it was and things have changed but I think I heard something about that being that case at least when concurrency first was added

kian•12/22/23, 1:40 AM

Nope.

someguy•12/23/23, 3:20 PM

I get this message but I am on a worker plan :/

Ssomeguy I get this message but I am on a worker plan :/

Matt Silverlock•12/23/23, 3:50 PM

What plan are you on? Can you share your account ID? It must be a paid plan.

MMatt Silverlock What plan are you on? Can you share your account ID? It must be a _paid_ plan.

someguy•12/23/23, 3:51 PM

3b391436ea162bcb7052d445e3908d7d I think is my account ID
here's my plan:

Ssomeguy 3b391436ea162bcb7052d445e3908d7d I think is my account ID here's my plan:

Matt Silverlock•12/23/23, 3:54 PM

When did you sign up/subscribe? Can you open a support ticket?

MMatt Silverlock When did you sign up/subscribe? Can you open a support ticket?

someguy•12/23/23, 3:54 PM

sure, I signed up probably a year ago for workers

Ssomeguy sure, I signed up probably a year ago for workers

Matt Silverlock•12/23/23, 3:55 PM

There's definitely a bug here - I don't see a proper subscription. Support is equipped to help dive in.

MMatt Silverlock There's definitely a bug here - I don't see a proper subscription. Support is eq...

someguy•12/23/23, 3:56 PM

Ticket ID: 3075268, thanks!

Caio Pinna•12/27/23, 2:54 PM

Any strategies to work with more concurrency? Our current open ai account supports 2M TPM but our jobs (running on queues) only can up to 60k/min at maximum concurrency

huw•12/29/23, 12:46 AM

I have a queue with multiple ‘topics’ (messages are segmented per user), and I want to run an item after all other messages for that user have been consumed. Right now I can add a ‘cleanup’ message after all of my other sendsends, and then pick the batch off the queue and run my cleanup code at the end of the batch, but if multiple batches are executing concurrently I can’t guarantee that my cleanup code will begin after all of the other messages have finished executing. Just wondering if anyone has any ideas for architecture here, or if this is something that’s just not possible without tracking & polling flags for message status.

Bram•1/3/24, 9:23 AM

Hi, I saw this in the changelog:

Developers building on Queues can now create up to 10,000 queues per account, enabling easier per-user, per-job and sharding use-cases.

In such use-case, how would bindings to a worker work?

BBram Hi, I saw this in the changelog: > Developers building on Queues can now create...

Matt Silverlock•1/3/24, 9:26 AM

You can reasonably bind up to ~5000 queues (or any other kind of resource) per Worker script today.

MMatt Silverlock You can reasonably bind up to ~5000 queues (or any other kind of resource) per W...

Bram•1/3/24, 9:31 AM

And you would then access it like so: env["some-queue-" + userId].send(...)env["some-queue-" + userId].send(...)?

BBram And you would then access it like so: `env["some-queue-" + userId].send(...)`?

Matt Silverlock•1/3/24, 9:34 AM

Yes, if that fits your use-case. Or bucket users into a smaller set of queues.

Aster•1/3/24, 9:35 AM

Hello !!
How can I track the queue size with grafana? :3

Edit: found https://developers.cloudflare.com/queues/learning/metrics/

Metrics · Cloudflare Queues

You can view the metrics for a Queue on your account via the Cloudflare dashboard . Navigate to Workers > Queues > your Queue and under the Metrics …

Bram•1/3/24, 9:36 AM

Yea ok, got it. Is partititoning eventually going to be built in or is that unlikely? Meaning you can simply set a partition key and it will take care of creating partitions (buckets) based on the configured amount of partitions where each partition can be consumed in parallel

Bram•1/3/24, 9:36 AM

Thanks for the quick responses btw!

quambo•1/3/24, 12:53 PM

Partitioning seems to be on their radar, but the focus is on throughput right now (Some updates for @Queues users:)

quambo•1/3/24, 12:57 PM

Would love to hear if there is more specific ideas/plans around partitioning from cf side, and if its somewhere upon the next features. Right now, we're using durableobjects to implement a paritioned queue, but we end up rebuilding a lot of the queues features (like retry, concurrency, batchsize, batch timeouts etc)

Qquambo Partitioning seems to be on their radar, but the focus is on throughput right no...

Bram•1/3/24, 2:54 PM

Thanks for the link, excited for HTTP Pull. Seems like partitioning is not coming anytime soon though.

Bram•1/7/24, 1:22 PM

Is there a way to derive the queue name from the binding?
I have a worker that consumes & produces to multiple queues, message.queuemessage.queue gives the name of the queue. I would like to avoid having to pass the queue names as additional variables (next to the bindings)

BBram Is there a way to derive the queue name from the binding? I have a worker that c...

Matt Silverlock•1/7/24, 3:20 PM

The MessageBatchMessageBatch type that contains the messages delivered to the consumer has the queue name: https://developers.cloudflare.com/queues/reference/javascript-apis/#messagebatch

Cloudflare Queues - JavaScript APIs · Cloudflare Queues

Cloudflare Queues is integrated with Cloudflare Workers. To send and receive messages, you must use a Worker.

Hey @Matt , how's the investigation going, do you have any updates on timelines? Should I just assum

Similar Threads