Cloudflare Developers•2y ago

We’ve discussed allowing batching by an object/JSON key, which would achieve this correctly. No tim

We’ve discussed allowing batching by an object/JSON key, which would achieve this correctly. No timeline yet.

Conner•2/29/24, 12:44 AM

Makes sense okay

Conner•2/29/24, 12:44 AM

Yeah the option for consumer delay / send delay is a bigger ask haha
Would love to see that

MMatt Silverlock We’ve discussed allowing batching by an object/JSON key, which would achieve thi...

Conner•2/29/24, 12:46 AM

Extra note on this though
For batching by parent key, would be amazing if another event was fired for when all child tasks all done
So could track success/fail of parent based on child tasks
Having to hit db everytime for that rn

Alisson Acioli•3/1/24, 11:48 AM

@Matt Is there a forecast for the delay in messages that failed to be rerun again?

Alex•3/1/24, 9:02 PM

Has anyone been having very long delays for messages to be picked up by consumers?

Matt SilverlockOP•3/4/24, 1:02 PM

FYI - this is now live: https://developers.cloudflare.com/queues/platform/changelog/#default-content-type-now-set-to-json

Cloudflare Docs

Changelog · Cloudflare Queues

The default content type for messages published to a queue is now json, which improves compatibility with the upcoming pull-based queues.

Uulrikstrid Is there any documentation for running queues locally, or at least using hosted ...

Vero•3/6/24, 3:10 PM

https://developers.cloudflare.com/queues/reference/local-development/

Cloudflare Docs

Local Development · Cloudflare Queues

Queues support local development workflows using Wrangler, the command-line interface for Workers. Wrangler runs the same version of Queues as …

alex•3/6/24, 6:52 PM

Is there some library for Go?

Sharkey•3/7/24, 12:28 AM

How has the general reliability of the beta been?

We are thinking of using CF queues to build our notification system off of in production. Is it production ready?

Prakash•3/7/24, 2:44 AM

I am facing issues while working on Queues locally, please help.

Prakash•3/7/24, 2:44 AM

Error

platform:dev: ✘ [ERROR] Uncaught (async) Error: Handler does not export a queue() function.
platform:dev: 
[wrangler-UserWorker:wrn] Dropped message "e1d974b8e338ba4bd883f45d1f630728" on queue "queue" after 3 failed attempts!

platform:dev: ✘ [ERROR] Uncaught (async) Error: Handler does not export a queue() function.
platform:dev: 
[wrangler-UserWorker:wrn] Dropped message "e1d974b8e338ba4bd883f45d1f630728" on queue "queue" after 3 failed attempts!

Uulrikstrid Can I run queue's locally when developing? From what I can tell I can't put thin...

Prakash•3/7/24, 2:47 AM

Same here, did you find any solution?

John Spurlock•3/9/24, 11:07 PM

excited to start adding queues to production workers once it hits GA - two things I'm curious about:

1. does adding a

Queue

Queue

environment binding limit the colos where the producing

fetch

fetch

handler can run, or change the behavior of the worker in any way?
2. does adding a

queue

queue

handler (consumer) in the same worker as an existing

fetch

fetch

handler limit where the

fetch

fetch

handler can run, or change the behavior of the fetch side of the worker in any way?

JJohn Spurlock excited to start adding queues to production workers once it hits GA - two thing...

Hello, I’m Allie!•3/10/24, 9:23 AM

1. No
2. No

HHello, I’m Allie!1. No 2. No

Matt SilverlockOP•3/10/24, 2:53 PM

Correct. The consumer (the queue() handler) is effectively its own entity. Not something you need to think about.

MMatt Silverlock Correct. The consumer (the queue() handler) is effectively its own entity. Not s...

John Spurlock•3/10/24, 2:53 PM

excellent, thanks!

DanGamble•3/10/24, 9:46 PM

If you retry a single message, does it go to the to the back of the queue or does it just get tried in the next batch?

Dalist•3/11/24, 9:17 AM

Hi everyone, is there any updates for this product yet?
https://developers.cloudflare.com/pub-sub/

Or is it built within Cloudflare Queues now?

Original message was deleted

Dalist•3/11/24, 9:20 AM

oh, didn't see that one, thank you @Leo

Uulrikstrid Is there a max time for how long time a consumer can take before it needs to be ...

John Spurlock•3/12/24, 4:15 PM

yes, it's 15 minutes

https://developers.cloudflare.com/queues/platform/limits/

switz•3/12/24, 7:34 PM

pull!

switz•3/12/24, 8:28 PM

question: with pull, will items sit in the queue for a long time if they are not pulled yet? whats the time limit?

zegevlier•3/12/24, 8:32 PM

Posting this here since it's relevant

https://fxtwitter.com/elithrar/status/1767629801500692709

FxTwitter / FixupX💬 4 🔁 1 ❤️ 23 👁️ 678

Matt Silverlock 🐀 (@elithrar)

Kicking the tires on a shiny new feature for @cloudflare Queues:

Sswitz question: with pull, will items sit in the queue for a long time if they are not...

zegevlier•3/12/24, 8:34 PM

Currently the limit is 4 days (see: https://developers.cloudflare.com/queues/platform/limits/)

Zzegevlier Posting this here since it's relevant 😄 https://fxtwitter.com/elithrar/status/...

switz•3/14/24, 5:35 PM

https://silverlock-queues-http-pull.cloudflare-docs-7ou.pages.dev/queues/reference/http-pull/

Cloudflare Docs

Cloudflare Queues - Pull consumers · Cloudflare Queues

A pull-based consumer allows you to pull from a queue over HTTP from any environment and/or programming language outside of Cloudflare Workers. A …

Jeroen•3/15/24, 2:43 AM

Is there no way to use cloudflare-queue without a worker (external service)? I am implementing an API that doesn't use HTTP, so I do not think I can use a worker for this.

switz•3/15/24, 3:48 AM

It’s coming. See the link above

Sswitz It’s coming. See the link above

Jeroen•3/15/24, 12:19 PM

That's to pull, I want to push

CanRau•3/15/24, 12:21 PM

from this thread it seems they're currently only working on pull Some updates for @Queues users:

JJeroen Is there no way to use cloudflare-queue without a worker (external service)? I a...

Matt SilverlockOP•3/15/24, 2:11 PM

Eventually, but you can stand up a very very small Worker to do that today. With HTTP Pull, there wasn't a way for folks to do that.

JJeroen Is there no way to use cloudflare-queue without a worker (external service)? I a...

Matt SilverlockOP•3/15/24, 2:12 PM

e.g. https://developers.cloudflare.com/queues/get-started/#write-your-producer-worker + any auth scheme you wish to implement (shared secret, pubkey, etc)

MMatt Silverlock e.g. https://developers.cloudflare.com/queues/get-started/#write-your-producer-w...

Jeroen•3/15/24, 2:13 PM

Thanks! I did consider this but it felt a bit over-engineered, but I guess I will have to reconsider it.

JJeroen Thanks! I did consider this but it felt a bit over-engineered, but I guess I wil...

Matt SilverlockOP•3/15/24, 4:38 PM

Example: https://silverlock-queues-write-to-q.cloudflare-docs-7ou.pages.dev/queues/examples/publish-to-a-queue-over-http/ - pretty short!

Cloudflare Docs

Queues - Publish Directly via HTTP · Cloudflare Queues

The following example shows you how to publsh messages to a queue from any HTTP client, using a shared secret to securely authenticate the client.

Old Man Maker•3/17/24, 10:35 AM

Hi - I have a worker that is the queue producer. Is in .toml and exposed through env.
This is a DO - intending to recieve a message from WS and forward that message into the queue.
I have set this up exactly the same as other workers that I have connected to queues - but this one does not recognise the queue types as shown in the image.
What am I missing here?

Old Man Maker•3/17/24, 10:41 AM

yeah - build types - not sure how my DO project is so far behind. Updated and all types now resolved.

OOld Man Maker Hi - I have a worker that is the queue producer. Is in .toml and exposed through...

Matt SilverlockOP•3/17/24, 7:16 PM

What version of workers-types are you using? That’s all that matters.

NickP•3/17/24, 8:01 PM

is it possible to setup a queue consumer from a Remix project on Cloudflare Pages?

NNickP is it possible to setup a queue consumer from a Remix project on Cloudflare Page...

forslunds•3/17/24, 10:21 PM

Not at pages at all. Create a worker that sends request to your app for now if you want it.

NickP•3/17/24, 11:08 PM

Is there a recommended way to orchestrate deploying multiple workers and pages at the same time that share common resources?

MMatt Silverlock What version of workers-types are you using? That’s all that matters.

Old Man Maker•3/18/24, 2:31 PM

Yeah was an old version - just looked awfully similar

louis•3/19/24, 3:55 PM

Hello everyone, planning to test the queues with a test project. Here the plan: I have a webapp which visualizes data from a D1 Database. The Database gets its data from a service which runs in a local network. Can I create a queue which populates the d1 data. I want to build a docker app which runs locally on that local network and pushes new data to that queue. Can I use the Cloudflare queues „outside of Cloudflare“?

We’ve discussed allowing batching by an object/JSON key, which would achieve this correctly. No tim

Similar Threads