Cloudflare Developers•2y ago

question: with pull, will items sit in the queue for a long time if they are not pulled yet? whats t

question: with pull, will items sit in the queue for a long time if they are not pulled yet? whats the time limit?

zegevlier•3/12/24, 8:32 PM

Posting this here since it's relevant

https://fxtwitter.com/elithrar/status/1767629801500692709

FxTwitter / FixupX💬 4 🔁 1 ❤️ 23 👁️ 678

Matt Silverlock 🐀 (@elithrar)

Kicking the tires on a shiny new feature for @cloudflare Queues:

Sswitz question: with pull, will items sit in the queue for a long time if they are not...

zegevlier•3/12/24, 8:34 PM

Currently the limit is 4 days (see: https://developers.cloudflare.com/queues/platform/limits/)

Zzegevlier Posting this here since it's relevant 😄 https://fxtwitter.com/elithrar/status/...

switzOP•3/14/24, 5:35 PM

https://silverlock-queues-http-pull.cloudflare-docs-7ou.pages.dev/queues/reference/http-pull/

Cloudflare Docs

Cloudflare Queues - Pull consumers · Cloudflare Queues

A pull-based consumer allows you to pull from a queue over HTTP from any environment and/or programming language outside of Cloudflare Workers. A …

Jeroen•3/15/24, 2:43 AM

Is there no way to use cloudflare-queue without a worker (external service)? I am implementing an API that doesn't use HTTP, so I do not think I can use a worker for this.

switzOP•3/15/24, 3:48 AM

It’s coming. See the link above

Sswitz It’s coming. See the link above

Jeroen•3/15/24, 12:19 PM

That's to pull, I want to push

CanRau•3/15/24, 12:21 PM

from this thread it seems they're currently only working on pull Some updates for @Queues users:

JJeroen Is there no way to use cloudflare-queue without a worker (external service)? I a...

Matt Silverlock•3/15/24, 2:11 PM

Eventually, but you can stand up a very very small Worker to do that today. With HTTP Pull, there wasn't a way for folks to do that.

JJeroen Is there no way to use cloudflare-queue without a worker (external service)? I a...

Matt Silverlock•3/15/24, 2:12 PM

e.g. https://developers.cloudflare.com/queues/get-started/#write-your-producer-worker + any auth scheme you wish to implement (shared secret, pubkey, etc)

MMatt Silverlock e.g. https://developers.cloudflare.com/queues/get-started/#write-your-producer-w...

Jeroen•3/15/24, 2:13 PM

Thanks! I did consider this but it felt a bit over-engineered, but I guess I will have to reconsider it.

JJeroen Thanks! I did consider this but it felt a bit over-engineered, but I guess I wil...

Matt Silverlock•3/15/24, 4:38 PM

Example: https://silverlock-queues-write-to-q.cloudflare-docs-7ou.pages.dev/queues/examples/publish-to-a-queue-over-http/ - pretty short!

Cloudflare Docs

Queues - Publish Directly via HTTP · Cloudflare Queues

The following example shows you how to publsh messages to a queue from any HTTP client, using a shared secret to securely authenticate the client.

Old Man Maker•3/17/24, 10:35 AM

Hi - I have a worker that is the queue producer. Is in .toml and exposed through env.
This is a DO - intending to recieve a message from WS and forward that message into the queue.
I have set this up exactly the same as other workers that I have connected to queues - but this one does not recognise the queue types as shown in the image.
What am I missing here?

Old Man Maker•3/17/24, 10:41 AM

yeah - build types - not sure how my DO project is so far behind. Updated and all types now resolved.

OOld Man Maker Hi - I have a worker that is the queue producer. Is in .toml and exposed through...

Matt Silverlock•3/17/24, 7:16 PM

What version of workers-types are you using? That’s all that matters.

NickP•3/17/24, 8:01 PM

is it possible to setup a queue consumer from a Remix project on Cloudflare Pages?

NNickP is it possible to setup a queue consumer from a Remix project on Cloudflare Page...

forslunds•3/17/24, 10:21 PM

Not at pages at all. Create a worker that sends request to your app for now if you want it.

NickP•3/17/24, 11:08 PM

Is there a recommended way to orchestrate deploying multiple workers and pages at the same time that share common resources?

MMatt Silverlock What version of workers-types are you using? That’s all that matters.

Old Man Maker•3/18/24, 2:31 PM

Yeah was an old version - just looked awfully similar

louis•3/19/24, 3:55 PM

Hello everyone, planning to test the queues with a test project. Here the plan: I have a webapp which visualizes data from a D1 Database. The Database gets its data from a service which runs in a local network. Can I create a queue which populates the d1 data. I want to build a docker app which runs locally on that local network and pushes new data to that queue. Can I use the Cloudflare queues „outside of Cloudflare“?

Amblamps•3/19/24, 4:44 PM

You’ll need to use a worker to interact with queues

Llouis Hello everyone, planning to test the queues with a test project. Here the plan: ...

Matt Silverlock•3/19/24, 9:33 PM

https://developers.cloudflare.com/queues/examples/publish-to-a-queue-over-http/ - you can write to a Queue, you just need to write a Worker to do that for you! The Worker can (of course) accept HTTP requests that write to the queue.

Cloudflare Docs

Queues - Publish Directly via HTTP · Cloudflare Queues

The following example shows you how to publsh messages to a queue from any HTTP client, using a shared secret to securely authenticate the client.

louis•3/20/24, 6:02 AM

So only one way in this case?

КАПИТАН ПСИНА (ВУФ)•3/20/24, 3:37 PM

Hi everyone.
I have a pages deployment that I am trying to add queues to and I ran into an issue with registering the worker there as a consumer.
I have a _worker.js that I have setup as a producer from settings in Cloudflare web interface, but there is no setting to bind it as a consumer. Is this possible at all? Or will I have to just create a separate worker that I can register as consumer via a toml file?

ККАПИТАН ПСИНА (ВУФ)Hi everyone. I have a pages deployment that I am trying to add queues to and I ...

Hello, I’m Allie!•3/20/24, 3:39 PM

You have to use a separate Worker. Pages Functions can only handle HTTP events, not Queues

HHello, I’m Allie!You have to use a separate Worker. Pages Functions can only handle HTTP events, ...

КАПИТАН ПСИНА (ВУФ)•3/20/24, 3:47 PM

Thanks!

Ggrimpa There is a possibility to a old Worker (Javascript) use Queues? Or the Worker ne...

Hello, I’m Allie!•3/21/24, 6:24 PM

Sure! TypeScript Workers are compiled down to JavaScript anyway

John Spurlock•3/21/24, 7:50 PM

Just got

Queue send failed: Queue is overloaded. Please back off.

Queue send failed: Queue is overloaded. Please back off.

for the first time. Does that only happen when the max per-queue message throughput is exceeded? (ie currently 400/s)

Or can it happen for other reasons?

Lloyd•3/22/24, 11:07 AM

How comsumer works if income message over 100per seconds?
case 1 or case 2?

Lloyd•3/22/24, 11:10 AM

oh i see. thankyou

JJohn Spurlock Just got `Queue send failed: Queue is overloaded. Please back off.` for the firs...

Matt Silverlock•3/22/24, 2:01 PM

You should get

Too Many Requests

Too Many Requests

when you hit the rate limit. Did you exceed your backlog in a single queue?

MMatt Silverlock You should get `Too Many Requests` when you hit the rate limit. Did you exceed y...

Matt Silverlock•3/22/24, 2:02 PM

MMatt Silverlock Click to see attachment

John Spurlock•3/22/24, 3:36 PM

curious - the backlog looked normal, and rates never get close to the limit (see weekly charts) - this is a constant process that's been running for over a week now with little variation.

Queue send failed: Queue is overloaded. Please back off.

Queue send failed: Queue is overloaded. Please back off.

was received from workers all over the world at around the same time (calling

.send

.send

on a queue).

Almost feels like an error message that would come from a DO request queue overload. Would it be possible for your team to track down what causes this message? I don't see any clues in workerd. Perhaps there can be an internal backup in whatever internal queue magic sits behind

.send

.send

and the queue storage?

Ideally from an abstraction pov,

.send

.send

would only fail when exceeding the msg/s limits (globally) or some truly edge case node failure where it couldn't even save to any sort of recoverable internal storage.

JJohn Spurlock curious - the backlog looked normal, and rates never get close to the limit (see...

Matt Silverlock•3/22/24, 3:38 PM

(queueId, accountId) - ? We'll take a look...

MMatt Silverlock (queueId, accountId) - ? We'll take a look...

Matt Silverlock•3/22/24, 3:39 PM

(I should just add yours to my keyboard text expander shortcuts...)

FFedera How are people looping the incoming batches from the queue?

John Spurlock•3/24/24, 3:51 PM

for (const msg of batch.messages) {

for (const msg of batch.messages) {

FFedera How are people looping the incoming batches from the queue?

Matt Silverlock•3/25/24, 1:04 PM

https://developers.cloudflare.com/queues/reference/javascript-apis/#consumer

Cloudflare Docs

Cloudflare Queues - JavaScript APIs · Cloudflare Queues

Cloudflare Queues is integrated with Cloudflare Workers. To send and receive messages, you must use a Worker.

Matt Silverlock•3/25/24, 5:48 PM

@Queues - You can now configure pull-based consumers (aka "HTTP Pull") in Queues! Pull-consumers can run outside of Cloudflare, pull + ack over HTTP, and scale out concurrently.

Example:

➜  curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" -X POST --data '{}' \
     -H "Authorization: Bearer ${QUEUES_TOKEN}" \
     -H "Content-Type:application/json"

➜  curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" -X POST --data '{}' \
     -H "Authorization: Bearer ${QUEUES_TOKEN}" \
     -H "Content-Type:application/json"

Docs: https://developers.cloudflare.com/queues/reference/pull-consumers/
Changelog: https://developers.cloudflare.com/queues/platform/changelog/#support-for-pull-based-consumers

switzOP•3/25/24, 9:42 PM

If anyone forks the SQS consumer package for Cloudflare queues let me know

switzOP•3/25/24, 9:43 PM

https://github.com/bbc/sqs-consumer

GitHub

GitHub - bbc/sqs-consumer: Build Amazon Simple Queue Service (SQS) ...

Build Amazon Simple Queue Service (SQS) based applications without the boilerplate - bbc/sqs-consumer