hello everyone I have a doubt related to
hello everyone I have a doubt related to the Cloudflare queues, so currently we host a platform where we automate multiple applications by receiving Webhooks on our platform.
Our domain is hosted via Cloudflare. Recently we have been facing downtime due to server maintenance. So I had thought of a approach that before going into downtime, we will enable a worker that would have a queue connected to it and in the worker we would have setup our route e.g. https://{{domian_name}}/* as soon as we enable this our system of the worker is up and captures webhooks and stores them into a queue.
The issue that I am facing is while processing the queues for around 300 webhooks the API is taking about 75 seconds to process those webhooks and send them to the respective destination.
I have tried them in sync with but still the time is approximately same.
Is there any way to process the queues faster??
Is there any better approach that I am not aware of.
Kindly do let me know.
3 Replies
So is pushing to the Queue taking 75 seconds?
Are you trying to push individual messages? Is pushing a batch feasible?
pushing into queue is not taking 75 seconds
I pushed lets say 500 webhooks
they get inserted into the queue instantly
but when i am pulling those out at that time the queue delivery is taking time
also for some reason some webhooks dont get delivered and they show up in the queue again after 5 minutes ( i kept visibility timeout to be 5 mins)
I just wanna know the best way to pull about webhooks 500K out of the queue and meanwhile having the least amount of webhooks for the retries.
So because you says you’re “pulling”, I imagine you have an HTTP consumer? In that case, are you able to control the concurrency? If so, you can pull as many batches concurrently as you want
The “messages showing up in the queue again after the visibility timeout” is expected, that’s basically what the visibility timeout means. The idea is that if one of your consumers crashes / gets stuck, you don’t want the message that it pulled (but didn’t ACK) to be able to be retried again after some time)