(queueId, accountId) - ? We'll take a look...

MMatt Silverlock (queueId, accountId) - ? We'll take a look...

M

Matt SilverlockOP•3/22/24, 3:39 PM

(I should just add yours to my keyboard text expander shortcuts...)

FFedera How are people looping the incoming batches from the queue?

J

John Spurlock•3/24/24, 3:51 PM

for (const msg of batch.messages) {

for (const msg of batch.messages) {

for (const msg of batch.messages) {

for (const msg of batch.messages) {

FFedera How are people looping the incoming batches from the queue?

M

Matt SilverlockOP•3/25/24, 1:04 PM

https://developers.cloudflare.com/queues/reference/javascript-apis/#consumer

Cloudflare Docs

Cloudflare Queues - JavaScript APIs · Cloudflare Queues

Cloudflare Queues is integrated with Cloudflare Workers. To send and receive messages, you must use a Worker.

M

Matt SilverlockOP•3/25/24, 5:48 PM

@Queues - You can now configure pull-based consumers (aka "HTTP Pull") in Queues! Pull-consumers can run outside of Cloudflare, pull + ack over HTTP, and scale out concurrently.

Example:

➜  curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" -X POST --data '{}' \
     -H "Authorization: Bearer ${QUEUES_TOKEN}" \
     -H "Content-Type:application/json"

➜  curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" -X POST --data '{}' \
     -H "Authorization: Bearer ${QUEUES_TOKEN}" \
     -H "Content-Type:application/json"

➜  curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" -X POST --data '{}' \
     -H "Authorization: Bearer ${QUEUES_TOKEN}" \
     -H "Content-Type:application/json"

➜  curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" -X POST --data '{}' \
     -H "Authorization: Bearer ${QUEUES_TOKEN}" \
     -H "Content-Type:application/json"

Docs: https://developers.cloudflare.com/queues/reference/pull-consumers/
Changelog: https://developers.cloudflare.com/queues/platform/changelog/#support-for-pull-based-consumers

S

switz•3/25/24, 9:42 PM

If anyone forks the SQS consumer package for Cloudflare queues let me know

S

switz•3/25/24, 9:43 PM

https://github.com/bbc/sqs-consumer

GitHub

GitHub - bbc/sqs-consumer: Build Amazon Simple Queue Service (SQS) ...

Build Amazon Simple Queue Service (SQS) based applications without the boilerplate - bbc/sqs-consumer

J

johncantrell97•3/26/24, 12:33 AM

any chance queues makes it way into the free plan at all? similar to D1?

C

cg•3/26/24, 3:21 AM

hey all

C

cg•3/26/24, 3:21 AM

am I able to get an assist from a CF team member to clear a Queue

C

cg•3/26/24, 3:21 AM

I have a backlog of 32.48M messages

C

cg•3/26/24, 5:35 AM

ah, I just deleted the queue

Z

zegevlier•3/26/24, 5:21 PM

I can't comment on that limit, but you could always temporarily store the message somewhere, then put an identifier in the queue and retrieve that identifier on the queue event.

Original message was deleted

M

Matt SilverlockOP•3/26/24, 6:03 PM

Fixed for now.

M

Matt SilverlockOP•3/26/24, 6:04 PM

@Queues We've just shipped the ability to delay messages - both when writing to a queue as well as when retrying. This can also be set at the queue and/or consumer level as a default delay that applies to all messages.

Sending:

// Delay a singular message by 600 seconds (10 minutes)
await env.YOUR_QUEUE.send(message, { delaySeconds: 600 })

// Delay a singular message by 600 seconds (10 minutes)
await env.YOUR_QUEUE.send(message, { delaySeconds: 600 })

// Delay a singular message by 600 seconds (10 minutes)
await env.YOUR_QUEUE.send(message, { delaySeconds: 600 })

// Delay a singular message by 600 seconds (10 minutes)
await env.YOUR_QUEUE.send(message, { delaySeconds: 600 })

Retrying:

export default {
  async queue(batch: MessageBatch, env: Env, ctx: ExecutionContext) {
    for (const msg of batch.messages) {
      // Mark for retry and delay a singular message
      // by 3600 seconds (1 hour)
      msg.retry({delaySeconds: 3600})

    }
  },
};

export default {
  async queue(batch: MessageBatch, env: Env, ctx: ExecutionContext) {
    for (const msg of batch.messages) {
      // Mark for retry and delay a singular message
      // by 3600 seconds (1 hour)
      msg.retry({delaySeconds: 3600})

    }
  },
};

export default {
  async queue(batch: MessageBatch, env: Env, ctx: ExecutionContext) {
    for (const msg of batch.messages) {
      // Mark for retry and delay a singular message
      // by 3600 seconds (1 hour)
      msg.retry({delaySeconds: 3600})

    }
  },
};

export default {
  async queue(batch: MessageBatch, env: Env, ctx: ExecutionContext) {
    for (const msg of batch.messages) {
      // Mark for retry and delay a singular message
      // by 3600 seconds (1 hour)
      msg.retry({delaySeconds: 3600})

    }
  },
};

Docs: https://developers.cloudflare.com/queues/reference/batching-retries/#delay-messages
Changelog: https://developers.cloudflare.com/queues/platform/changelog/#delay-messages-published-to-a-queue

Original message was deleted

M

Matt SilverlockOP•3/26/24, 6:04 PM

You can also split the message and write a batch, or write to R2 and use the upcoming Event Notifications feature to trigger an "event" when you write to R2 into a Queue so you can post-process the data.

Rrelero90 I am running into a challenge with TypeScript types and setting up a Queue produ...

M

Matt SilverlockOP•3/26/24, 6:14 PM

add()

add()

is not a method on a Queue. Where are you seeing this?

add()

add()

is not mentioned in the tutorial.

send()

send()

is what you want to call.

Tutorial: https://developers.cloudflare.com/queues/get-started/
API docs: https://developers.cloudflare.com/queues/reference/javascript-apis/#producer

Cloudflare Docs

Get started · Cloudflare Queues

Cloudflare Queues is a flexible messaging queue that allows you to queue messages for asynchronous processing. By following this guide, you will …

Cloudflare Docs

Cloudflare Queues - JavaScript APIs · Cloudflare Queues

Cloudflare Queues is integrated with Cloudflare Workers. To send and receive messages, you must use a Worker.

Original message was deleted

M

Matt SilverlockOP•3/26/24, 7:20 PM

Good things take time

We'll have more updates to share publicly next week.

Original message was deleted

M

Matt SilverlockOP•3/26/24, 8:47 PM

Not at the moment, no.

U

Unsmart•3/26/24, 9:16 PM

I'd hope your payment processor would have deduplication on their end...

U

Unsmart•3/26/24, 9:51 PM

[[env.staging.queues.producers]]

Original message was deleted

W

Walshy•3/26/24, 9:54 PM

Yeah you'll need [[env.staging.queue.producers]]

W

Walshy•3/26/24, 9:54 PM

TOML is not a great config language imo :meowupsidedown:

FFlorian That's a starting point! There have been multiple discussions around delays and ...

M

Matt SilverlockOP•3/27/24, 1:01 AM

You can apply an exponential back off on each attempt - we’ll soon be exposing the # of attempts as part of the message metadata. With that and the ability to set a delay you can write your own back off alg.

V

Veld•3/27/24, 7:06 PM

oh dang

V

Veld•3/27/24, 7:06 PM

cloudflare to aws any% speedrun

K

kamama•3/28/24, 6:53 AM

Hi folks, is queue not supporting multiple workers in local?
lets say
worker1 : producer
worker2: consumer?

Kkamama Hi folks, is queue not supporting multiple workers in local? lets say worker1 : ...

M

Matt SilverlockOP•3/28/24, 4:09 PM

https://developers.cloudflare.com/queues/reference/local-development/#known-issues

Cloudflare Docs

Local Development · Cloudflare Queues

Queues support local development workflows using Wrangler, the command-line interface for Workers. Wrangler runs the same version of Queues as …

Original message was deleted

M

Matt SilverlockOP•3/29/24, 2:44 PM

https://developers.cloudflare.com/queues/reference/batching-retries/#delay-messages

Cloudflare Docs

Batching and Retries · Cloudflare Queues

When configuring a consumer Worker for a queue, you can also define how messages are batched as they are delivered.

MMatt Silverlock https://developers.cloudflare.com/queues/reference/batching-retries/#delay-messa...

M

Matt SilverlockOP•3/29/24, 2:45 PM

S

switz•3/29/24, 9:27 PM

@Matt interestingly, looks like SQS will wait a short amount of time before responding if there are no messages (

 Pull consumers are designed to use a "short polling" approach, this means that the API from CloudFlare will respond immediately with any messages that are available, or an empty response if there are no messages available, this is different from SQS will wait an amount of time before responding with an empty response.

 Pull consumers are designed to use a "short polling" approach, this means that the API from CloudFlare will respond immediately with any messages that are available, or an empty response if there are no messages available, this is different from SQS will wait an amount of time before responding with an empty response.

 Pull consumers are designed to use a "short polling" approach, this means that the API from CloudFlare will respond immediately with any messages that are available, or an empty response if there are no messages available, this is different from SQS will wait an amount of time before responding with an empty response.

 Pull consumers are designed to use a "short polling" approach, this means that the API from CloudFlare will respond immediately with any messages that are available, or an empty response if there are no messages available, this is different from SQS will wait an amount of time before responding with an empty response.

). is that something that http pull would benefit from here? or are you of the opinion that fast-response > quantity of requests is more important.

S

switz•3/29/24, 9:28 PM

nicholas is making a lot of progress over at https://github.com/bbc/cloudflare-queue-consumer

can't wait to start using it as it stabilizes

GitHub

GitHub - bbc/cloudflare-queue-consumer: Build CloudFlare Queue base...

Build CloudFlare Queue based applications without the boilerplate (based on SQS Consumer) - bbc/cloudflare-queue-consumer

Sswitz @Matt interestingly, looks like SQS will wait a short amount of time before resp...

M

Matt SilverlockOP•3/29/24, 10:06 PM

SQS supports two modes: long and short. Right now we support what’s typically known as short - which is more aligned with the idea of “pulling” when you’re ready. TBD on whether long pulling makes sense vs other work.

S

switz•3/29/24, 10:30 PM

another request, perhaps should be over at #r2, but would be nice to send r2 lifecycle events to a cloudflare queue

S

switz•3/29/24, 10:31 PM

https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html

Amazon S3 Event Notifications - Amazon Simple Storage Service

Set up and configure notifications so that key events on buckets cause a message to be sent to an Amazon SNS topic.

M

mau•3/30/24, 11:11 AM

Trying to implement queues in nextjs, not able to find any example. Does anyone has any poc?

Mmau Trying to implement queues in nextjs, not able to find any example. Does anyone ...

M

Matt SilverlockOP•3/30/24, 2:15 PM

https://developers.cloudflare.com/pages/framework-guides/nextjs/deploy-a-nextjs-site/#use-bindings-in-your-nextjs-application - replace D1/KV with Queues here.

Cloudflare Docs

Full-stack deployment · Cloudflare Pages docs

Deploy a full-stack Next.js site (recommended).

M

mau•3/30/24, 5:55 PM

Thanks @Matt for reply. I am not able to find the code implementation for nextjs. I did figure out the bindings part. For example:
async queue(){} there is nothing like that in nextjs… so how to hook that in nextjs. Thanks

Mmau Thanks @Matt for reply. I am not able to find the code implementation for nextjs...

M

Matt SilverlockOP•3/30/24, 6:02 PM

Your queue consumer won’t be part of your Next.js app - that’s not how your app would work. You write to the queue from your Next.js app. This isn’t a limitation of Queues - your Next.js app doesn’t have a concept of a “queue consumer” here since it’s request-response driven.

What are you trying to use a queue for here?

M

mau•3/31/24, 5:07 AM

I am developing an e-learning application and focusing on optimizing data processing. Our system evaluates quiz answers in real-time, providing immediate feedback. We store the quiz progress and answers in MongoDB, allowing students to resume their work without any issues.However, we face a challenge with the delay users experience at the result screen. This delay occurs because, currently, when a student submits a quiz, we transfer the data from MongoDB to PostgreSQL. This data transfer and processing happen in the same route as the quiz submission, leading to a wait time for the user to see their results. To mitigate this, we are considering implementing queues to handle the data transfer process separately, thereby reducing the load during submission and speeding up the display of results. Additionally, we are exploring the use of key-value stores or durable objects to further optimize this workflow.

M

mau•3/31/24, 5:08 AM

We are trying to use all the products provided by Cloudflare so that we can build highly Scaleable product.

Mmau I am developing an e-learning application and focusing on optimizing data proces...

M

Matt SilverlockOP•3/31/24, 3:33 PM

It’s hard to fully understand your architecture at this level, but if you want to make your application more real time, I’m not sure how Queues would do that. Using a Durable Object or just D1 would likely be better - no need to “sync” or wait on data? It’s a quiz. It should be immediate.

M

mau•3/31/24, 3:36 PM

Result will be still immediate, it’s just I was thinking to build a asynchronous way of moving data from mongo (which is used to store data while users are working on the quiz) to Postgres’s (which is final destination)… I’m also exploring d1 (because I need something which allows unstructured data and it’s quick which will replace mongo).

U

Unsmart•3/31/24, 5:54 PM

I’m confused why use multiple databases like that for the same data

Mmau Result will be still immediate, it’s just I was thinking to build a asynchronous...

M

Matt SilverlockOP•3/31/24, 10:02 PM

Right, if the user response does NOT need to block on that data movement, then Queues can be used as a way to batch up changes and then write them to Postgres from your consumer.

You would bind to the Queue and call

send()

send()

within your Next.js app exactly like the Pages tutorial I sent you.

Your consumer - the part that consumes the messages and writes to Postgres - would just be a standalone Worker.

(queueId, accountId) - ? We'll take a look...

Similar Threads

Similar Threads

Similar Threads