Cloudflare Developers•2y ago

If an ack fails, should it be retried in the same block, or would it have failed for the entire batc

If an ack fails, should it be retried in the same block, or would it have failed for the entire batch run?

paddebal•6/28/24, 10:10 PM

did someone really tag everyone here

paddebal•6/28/24, 10:10 PM

smh

HHello, I’m Allie!If an ack fails, should it be retried in the same block, or would it have failed...

Matt Silverlock•6/28/24, 11:11 PM

It’s possible for a single ack() to fail (rare as it may be)

Milo•7/1/24, 1:34 PM

Regarding the Queues REST API, I'm looking at the docs for the 'Pull Queue Messages' endpoint and documented there is the result_inforesult_info field.

This field would be quite useful for me because I'd like to see the queue backlog size, but it isn't actually returned by the API. I noticed it doesn't have RequiredRequired next to it, but is there a way to enable returning that field or does it just not exist?

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

MMilo Regarding the Queues REST API, I'm looking at [the docs for the 'Pull Queue Mess...

Matt Silverlock•7/1/24, 1:59 PM

result_inforesult_info is for paginated results - not for backlogs. We don’t currently return the backlog count in the response at all.

Milo•7/1/24, 2:00 PM

Thanks

Ben•7/1/24, 2:49 PM

Hey for some reason my deployed queues are simply not consuming the queues.

I get no logs, no errors, they just dont process, any tips for debugging this?

actually I think I think I fixed it xD

rewbs•7/1/24, 2:53 PM

Hi all, I'm working with an R2 event notification queue and everything is working smoothly, love the feature and ease of use. I was wondering if there's any chance object metadata might be included in the upload notification event payload in the future?

PPranshu Maheshwari Thanks for highlighting this, we're looking into it

carlos-alberto•7/1/24, 8:10 PM

Hey @Pranshu Maheshwari , was there any update on our issue?

Ccarlos-alberto Hey @Pranshu Maheshwari , was there any update on our issue?

Pranshu Maheshwari•7/1/24, 10:19 PM

We noticed networking issues with a few queues last week. We're still investigating the root cause, but some of the queues that were returning 500s are working fine now. It's one of the issues we're ironing out while queues is in beta!

Shivek•7/2/24, 11:06 AM

Hey, is there no way to work with queues locally?

Shivek•7/2/24, 11:06 AM

I'm building an app using KV and Queues. I have set remote=true so I can read env variables from dev workers env, and use dev kv bucket.

Shivek•7/2/24, 11:07 AM

But when I introduced Queues, I got this warning and my app breaks down:

▲ [WARNING] Queues are currently in Beta and are not supported in wrangler dev remote mode.

▲ [WARNING] Queues are currently in Beta and are not supported in wrangler dev remote mode.

SShivek Hey, is there no way to work with queues locally?

Hard@Work•7/2/24, 11:24 AM

Sort of. As you noted, you can use them locally, but then you can't connect to any remote resources

HHard@Work Sort of. As you noted, you can use them locally, but then you can't connect to a...

Shivek•7/2/24, 11:36 AM

So I turned off remote=true and injected my env variables as .dev.vars. But even then I'm not able to use queues.

Shivek•7/2/24, 11:37 AM

  try {
    console.log(1);
    const q = queue.factory(c);
    console.log(1.2);
    await q.send(item);
    console.log(2);
  } catch (error) {
    console.log(3);
    console.log(error);
  }

  try {
    console.log(1);
    const q = queue.factory(c);
    console.log(1.2);
    await q.send(item);
    console.log(2);
  } catch (error) {
    console.log(3);
    console.log(error);
  }

Some sample code. q.sendq.send fails silently. 2 is never printed and I get no errors.

SShivek ```ts try { console.log(1); const q = queue.factory(c); console.lo...

Hard@Work•7/2/24, 11:37 AM

Both the Producer and Consumer must be the same Worker

Hard@Work•7/2/24, 11:37 AM

At least for now

Hard@Work•7/2/24, 11:38 AM

(iirc)

HHard@Work Both the Producer and Consumer must be the same Worker

Shivek•7/2/24, 11:39 AM

I have no consumer setup right now. I'm just trying to see if send is working

Shivek•7/2/24, 11:39 AM

it is not at the moment

SShivek I have no consumer setup right now. I'm just trying to see if send is working

Hard@Work•7/2/24, 11:39 AM

Yeah, it might fail if there is nothing to write to, I'm not sure

HHard@Work Yeah, it *might* fail if there is nothing to write to, I'm not sure

Shivek•7/2/24, 11:45 AM

My config seems correct and I see the resource being attached when I run wrangler dev

Shivek•7/2/24, 11:45 AM

Your worker has access to the following bindings:
- KV Namespaces:
  - xxxx: xxxx
- Queues:
  - TREASURY_QUEUE: dev-treasury-queue

Your worker has access to the following bindings:
- KV Namespaces:
  - xxxx: xxxx
- Queues:
  - TREASURY_QUEUE: dev-treasury-queue

SShivek ```Your worker has access to the following bindings: - KV Namespaces: - xxxx: ...

Hard@Work•7/2/24, 11:45 AM

Yes, but if there isn't a consumer for that queue, I would guess it might still fail

Shivek•7/2/24, 11:46 AM

okay, so both producer and consumers need to be defined before I can write?

SShivek okay, so both producer and consumers need to be defined before I can write?

Hard@Work•7/2/24, 11:47 AM

Maybe? Tbh, I find it odd that it isn't erroring at all, but just silently failing. If you are able to wait a bit, I may be able to test it out and see if there is something else going wrong

HHard@Work Maybe? Tbh, I find it odd that it isn't erroring at all, but just silently faili...

Shivek•7/2/24, 11:48 AM

I can wait, I'll try to set a consumer in the meanwhile

Shivek•7/2/24, 12:00 PM

@Hard@Work | R2 I'm able to send messages if a consumer is registered. I was assuming that if there are no consumers, the messages will just stay in the Queue. That's not the case. If there are no consumers, the Q didn't register any messages.

PPranshu Maheshwari Could you fill out this form?https://docs.google.com/forms/d/e/1FAIpQLSd_fwAVObo...

Michael Wittig•7/3/24, 8:40 AM

Hi! I received no response so far after filling out the form. How long does it usually take?

johtso•7/3/24, 2:55 PM

Are retries supposed to work when testing locally?

Jjohtso Are retries supposed to work when testing locally?

Pranshu Maheshwari•7/3/24, 9:25 PM

yes! could you share your code if it's not working on your side?

PPranshu Maheshwari yes! could you share your code if it's not working on your side?

johtso•7/3/24, 9:29 PM

ok good to know, let me try and make a super minimal example and double check

johtso•7/3/24, 9:32 PM

ok, that seems to be working!

johtso•7/3/24, 9:35 PM

probably what I was noticing was messages not surviving between dev server reloads

letincho5•7/4/24, 10:49 AM

Hi... I need a recommendation. To improve or maximize performance (message processing per second), which is better: 1 worker with 10 queues (and consumer X in each queue) or 10 workers with 1 queue each, with consumer X in each queue?

mlshvdv•7/4/24, 11:35 AM

@Pranshu Maheshwari Hi! Some questions about Queues:

1. Do you plan to support message deduplication (refusing to publish the same message while it exists in the queue + support for 'force' mode to rewrite it)?
2. Do you have any plans to add an API method to be able to remove a message from the queue by its ID or impedance key?
3. Are you planning to change the maximum delay to 1 year (this is supported by Upstash QStash - very convenient)?
4. Is there an estimated release date for Queues in production?

Thanks!

control_crusher•7/5/24, 3:46 PM

Does anyone have example code of how they handled multiple queues, single worker?

Sounds like, bind multiple in repeat bind blocks. Then switch inside the Queue handler based on which queue the messages came from.

Follow-up question... do batches then include messages from any of the bound queues? If so, do we only define the batch settings in a single bind block? @Pranshu Maheshwari

Ccontrol_crusher Does anyone have example code of how they handled multiple queues, single worker...

Matt Silverlock•7/6/24, 1:04 PM

Producers are entirely separate from consumers.

- You can write to any queues you bind to as a producer.
- Your consumer will receive messages from only the queues it is configured as a consumer for.
- messages will be batched by queue, but each batch can be from any of the queues you have bound to.

Matt Silverlock•7/6/24, 1:05 PM

https://developers.cloudflare.com/queues/configuration/javascript-apis/#consumer and https://developers.cloudflare.com/queues/reference/how-queues-works/#create-a-consumer-worker

Ben•7/8/24, 5:01 PM

When queues go out of beta will they be available under workers free plan?

BBen When queues go out of beta will they be available under workers free plan?

Pranshu Maheshwari•7/8/24, 6:54 PM

we intend to make queues available under the free plan (with limits of course) in the future, although we don't have a timeline yet

PPranshu Maheshwari we intend to make queues available under the free plan (with limits of course) i...

Ben•7/8/24, 6:54 PM

sounds great!

BumblebeeSquare•7/8/24, 8:22 PM

is there message group like feature in queue ? i.e sqs message group

Brett Willis•7/9/24, 4:25 AM

It would be great to be able to manually pause/resume queues (collect from producers but pause dispatch to consumers) and also to purge queue messages. This help both in development and in production e.g. maintenance/migration, disaster recovery, or when dependent services are experiencing downtime. Queues are good for decoupling services for this very reason, and it's good to be able to intervene in emergencies!

minty•7/9/24, 8:30 PM

is it possible to consume messages from a queue from a non-worker piece of code? i.e. the way you can consume messages from SQS using their SDK in the language of your choice via polling/fetching?

Mminty is it possible to consume messages from a queue from a non-worker piece of code?...

Hello, I’m Allie!OP•7/9/24, 8:37 PM

Yes, see: https://developers.cloudflare.com/queues/configuration/pull-consumers

Cloudflare Docs

Cloudflare Queues - Pull consumers · Cloudflare Queues

A pull-based consumer allows you to pull from a queue over HTTP from any environment and/or programming language outside of Cloudflare Workers. A …

daguang•7/11/24, 10:53 AM

@Hard@Work @HardlyWorkin' Hello, when I use R2 GET data when consumers use R2 GET data, it will cause the queue to complete it directly. I tried to use Try Catch to capture the error, but I found nothing. Is this my configuration file wrong? or a bug ? (And this logic is normal in the normal worker fetch)

kian•7/11/24, 4:41 PM

you're returning errors rather than throwing them?

kian•7/11/24, 4:41 PM

:meowthinking:

If an ack fails, should it be retried in the same block, or would it have failed for the entire batc

Similar Threads