is it possible to setup a queue consumer from a Remix project on Cloudflare Pages?

NNickP is it possible to setup a queue consumer from a Remix project on Cloudflare Page...

F

forslunds•3/17/24, 10:21 PM

Not at pages at all. Create a worker that sends request to your app for now if you want it.

N

NickPOP•3/17/24, 11:08 PM

Is there a recommended way to orchestrate deploying multiple workers and pages at the same time that share common resources?

MMatt Silverlock What version of workers-types are you using? That’s all that matters.

O

Old Man Maker•3/18/24, 2:31 PM

Yeah was an old version - just looked awfully similar

L

louis•3/19/24, 3:55 PM

Hello everyone, planning to test the queues with a test project. Here the plan: I have a webapp which visualizes data from a D1 Database. The Database gets its data from a service which runs in a local network. Can I create a queue which populates the d1 data. I want to build a docker app which runs locally on that local network and pushes new data to that queue. Can I use the Cloudflare queues „outside of Cloudflare“?

A

Amblamps•3/19/24, 4:44 PM

You’ll need to use a worker to interact with queues

Llouis Hello everyone, planning to test the queues with a test project. Here the plan: ...

M

Matt Silverlock•3/19/24, 9:33 PM

https://developers.cloudflare.com/queues/examples/publish-to-a-queue-over-http/ - you can write to a Queue, you just need to write a Worker to do that for you! The Worker can (of course) accept HTTP requests that write to the queue.

Cloudflare Docs

Queues - Publish Directly via HTTP · Cloudflare Queues

The following example shows you how to publsh messages to a queue from any HTTP client, using a shared secret to securely authenticate the client.

L

louis•3/20/24, 6:02 AM

So only one way in this case?

К

КАПИТАН ПСИНА (ВУФ)•3/20/24, 3:37 PM

Hi everyone.
I have a pages deployment that I am trying to add queues to and I ran into an issue with registering the worker there as a consumer.
I have a _worker.js that I have setup as a producer from settings in Cloudflare web interface, but there is no setting to bind it as a consumer. Is this possible at all? Or will I have to just create a separate worker that I can register as consumer via a toml file?

ККАПИТАН ПСИНА (ВУФ)Hi everyone. I have a pages deployment that I am trying to add queues to and I ...

H

Hello, I’m Allie!•3/20/24, 3:39 PM

You have to use a separate Worker. Pages Functions can only handle HTTP events, not Queues

HHello, I’m Allie!You have to use a separate Worker. Pages Functions can only handle HTTP events, ...

К

КАПИТАН ПСИНА (ВУФ)•3/20/24, 3:47 PM

Thanks!

Ggrimpa There is a possibility to a old Worker (Javascript) use Queues? Or the Worker ne...

H

Hello, I’m Allie!•3/21/24, 6:24 PM

Sure! TypeScript Workers are compiled down to JavaScript anyway

J

John Spurlock•3/21/24, 7:50 PM

Just got

Queue send failed: Queue is overloaded. Please back off.

Queue send failed: Queue is overloaded. Please back off.

for the first time. Does that only happen when the max per-queue message throughput is exceeded? (ie currently 400/s)

Or can it happen for other reasons?

L

Lloyd•3/22/24, 11:07 AM

How comsumer works if income message over 100per seconds?
case 1 or case 2?

L

Lloyd•3/22/24, 11:10 AM

oh i see. thankyou

JJohn Spurlock Just got `Queue send failed: Queue is overloaded. Please back off.` for the firs...

M

Matt Silverlock•3/22/24, 2:01 PM

You should get

Too Many Requests

Too Many Requests

when you hit the rate limit. Did you exceed your backlog in a single queue?

MMatt Silverlock You should get `Too Many Requests` when you hit the rate limit. Did you exceed y...

M

Matt Silverlock•3/22/24, 2:02 PM

MMatt Silverlock Click to see attachment

J

John Spurlock•3/22/24, 3:36 PM

curious - the backlog looked normal, and rates never get close to the limit (see weekly charts) - this is a constant process that's been running for over a week now with little variation.

Queue send failed: Queue is overloaded. Please back off.

Queue send failed: Queue is overloaded. Please back off.

was received from workers all over the world at around the same time (calling

.send

.send

on a queue).

Almost feels like an error message that would come from a DO request queue overload. Would it be possible for your team to track down what causes this message? I don't see any clues in workerd. Perhaps there can be an internal backup in whatever internal queue magic sits behind

.send

.send

and the queue storage?

Ideally from an abstraction pov,

.send

.send

would only fail when exceeding the msg/s limits (globally) or some truly edge case node failure where it couldn't even save to any sort of recoverable internal storage.

JJohn Spurlock curious - the backlog looked normal, and rates never get close to the limit (see...

M

Matt Silverlock•3/22/24, 3:38 PM

(queueId, accountId) - ? We'll take a look...

MMatt Silverlock (queueId, accountId) - ? We'll take a look...

M

Matt Silverlock•3/22/24, 3:39 PM

(I should just add yours to my keyboard text expander shortcuts...)

FFedera How are people looping the incoming batches from the queue?

J

John Spurlock•3/24/24, 3:51 PM

for (const msg of batch.messages) {

for (const msg of batch.messages) {

for (const msg of batch.messages) {

for (const msg of batch.messages) {

FFedera How are people looping the incoming batches from the queue?

M

Matt Silverlock•3/25/24, 1:04 PM

https://developers.cloudflare.com/queues/reference/javascript-apis/#consumer

Cloudflare Docs

Cloudflare Queues - JavaScript APIs · Cloudflare Queues

Cloudflare Queues is integrated with Cloudflare Workers. To send and receive messages, you must use a Worker.

M

Matt Silverlock•3/25/24, 5:48 PM

@Queues - You can now configure pull-based consumers (aka "HTTP Pull") in Queues! Pull-consumers can run outside of Cloudflare, pull + ack over HTTP, and scale out concurrently.

Example:

➜  curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" -X POST --data '{}' \
     -H "Authorization: Bearer ${QUEUES_TOKEN}" \
     -H "Content-Type:application/json"

➜  curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" -X POST --data '{}' \
     -H "Authorization: Bearer ${QUEUES_TOKEN}" \
     -H "Content-Type:application/json"

➜  curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" -X POST --data '{}' \
     -H "Authorization: Bearer ${QUEUES_TOKEN}" \
     -H "Content-Type:application/json"

➜  curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" -X POST --data '{}' \
     -H "Authorization: Bearer ${QUEUES_TOKEN}" \
     -H "Content-Type:application/json"

Docs: https://developers.cloudflare.com/queues/reference/pull-consumers/
Changelog: https://developers.cloudflare.com/queues/platform/changelog/#support-for-pull-based-consumers

S

switz•3/25/24, 9:42 PM

If anyone forks the SQS consumer package for Cloudflare queues let me know

S

switz•3/25/24, 9:43 PM

https://github.com/bbc/sqs-consumer

GitHub

GitHub - bbc/sqs-consumer: Build Amazon Simple Queue Service (SQS) ...

Build Amazon Simple Queue Service (SQS) based applications without the boilerplate - bbc/sqs-consumer

J

johncantrell97•3/26/24, 12:33 AM

any chance queues makes it way into the free plan at all? similar to D1?

C

cg•3/26/24, 3:21 AM

hey all

C

cg•3/26/24, 3:21 AM

am I able to get an assist from a CF team member to clear a Queue

C

cg•3/26/24, 3:21 AM

I have a backlog of 32.48M messages

C

cg•3/26/24, 5:35 AM

ah, I just deleted the queue

Z

zegevlier•3/26/24, 5:21 PM

I can't comment on that limit, but you could always temporarily store the message somewhere, then put an identifier in the queue and retrieve that identifier on the queue event.

Original message was deleted

M

Matt Silverlock•3/26/24, 6:03 PM

Fixed for now.

M

Matt Silverlock•3/26/24, 6:04 PM

@Queues We've just shipped the ability to delay messages - both when writing to a queue as well as when retrying. This can also be set at the queue and/or consumer level as a default delay that applies to all messages.

Sending:

// Delay a singular message by 600 seconds (10 minutes)
await env.YOUR_QUEUE.send(message, { delaySeconds: 600 })

// Delay a singular message by 600 seconds (10 minutes)
await env.YOUR_QUEUE.send(message, { delaySeconds: 600 })

// Delay a singular message by 600 seconds (10 minutes)
await env.YOUR_QUEUE.send(message, { delaySeconds: 600 })

// Delay a singular message by 600 seconds (10 minutes)
await env.YOUR_QUEUE.send(message, { delaySeconds: 600 })

Retrying:

export default {
  async queue(batch: MessageBatch, env: Env, ctx: ExecutionContext) {
    for (const msg of batch.messages) {
      // Mark for retry and delay a singular message
      // by 3600 seconds (1 hour)
      msg.retry({delaySeconds: 3600})

    }
  },
};

export default {
  async queue(batch: MessageBatch, env: Env, ctx: ExecutionContext) {
    for (const msg of batch.messages) {
      // Mark for retry and delay a singular message
      // by 3600 seconds (1 hour)
      msg.retry({delaySeconds: 3600})

    }
  },
};

export default {
  async queue(batch: MessageBatch, env: Env, ctx: ExecutionContext) {
    for (const msg of batch.messages) {
      // Mark for retry and delay a singular message
      // by 3600 seconds (1 hour)
      msg.retry({delaySeconds: 3600})

    }
  },
};

export default {
  async queue(batch: MessageBatch, env: Env, ctx: ExecutionContext) {
    for (const msg of batch.messages) {
      // Mark for retry and delay a singular message
      // by 3600 seconds (1 hour)
      msg.retry({delaySeconds: 3600})

    }
  },
};

Docs: https://developers.cloudflare.com/queues/reference/batching-retries/#delay-messages
Changelog: https://developers.cloudflare.com/queues/platform/changelog/#delay-messages-published-to-a-queue

Original message was deleted

M

Matt Silverlock•3/26/24, 6:04 PM

You can also split the message and write a batch, or write to R2 and use the upcoming Event Notifications feature to trigger an "event" when you write to R2 into a Queue so you can post-process the data.

Rrelero90 I am running into a challenge with TypeScript types and setting up a Queue produ...

M

Matt Silverlock•3/26/24, 6:14 PM

add()

add()

is not a method on a Queue. Where are you seeing this?

add()

add()

is not mentioned in the tutorial.

send()

send()

is what you want to call.

Tutorial: https://developers.cloudflare.com/queues/get-started/
API docs: https://developers.cloudflare.com/queues/reference/javascript-apis/#producer

Cloudflare Docs

Get started · Cloudflare Queues

Cloudflare Queues is a flexible messaging queue that allows you to queue messages for asynchronous processing. By following this guide, you will …

Cloudflare Docs

Cloudflare Queues - JavaScript APIs · Cloudflare Queues

Cloudflare Queues is integrated with Cloudflare Workers. To send and receive messages, you must use a Worker.

Original message was deleted

M

Matt Silverlock•3/26/24, 7:20 PM

Good things take time

We'll have more updates to share publicly next week.

Original message was deleted

M

Matt Silverlock•3/26/24, 8:47 PM

Not at the moment, no.

U

Unsmart•3/26/24, 9:16 PM

I'd hope your payment processor would have deduplication on their end...

U

Unsmart•3/26/24, 9:51 PM

[[env.staging.queues.producers]]

Original message was deleted

W

Walshy•3/26/24, 9:54 PM

Yeah you'll need [[env.staging.queue.producers]]

W

Walshy•3/26/24, 9:54 PM

TOML is not a great config language imo :meowupsidedown:

FFlorian That's a starting point! There have been multiple discussions around delays and ...

M

Matt Silverlock•3/27/24, 1:01 AM

You can apply an exponential back off on each attempt - we’ll soon be exposing the # of attempts as part of the message metadata. With that and the ability to set a delay you can write your own back off alg.

V

Veld•3/27/24, 7:06 PM

oh dang

V

Veld•3/27/24, 7:06 PM

cloudflare to aws any% speedrun

is it possible to setup a queue consumer from a Remix project on Cloudflare Pages?

Similar Threads

Similar Threads

Similar Threads