https://developers.cloudflare.com/queues/learning/local-development/#known-issues This is a blocker

A

arbfay•10/18/23, 12:48 PM

Error: Queue send failed: Service Temporarily Unavailable

Error: Queue send failed: Service Temporarily Unavailable

Error: Queue send failed: Service Temporarily Unavailable

Error: Queue send failed: Service Temporarily Unavailable ??

A

arbfay•10/18/23, 12:51 PM

What is the service level for queues? Send failure rates?

CCharlie Its on the roadmap 😄

V

vassbence•10/18/23, 6:29 PM

Is it a goal before the product leaves beta?

MMatt Silverlock This is coming (HTTP Pull) in the future.

S

switz•10/18/23, 7:15 PM

any idea when this is coming, it's basically the only thing keeping me on SQS

I

iustin.pericica•10/18/23, 8:08 PM

Hi there,
We have a project based on cloudflare workers & queues. We currently have 1 queue with this config ( batch size:50, wait time: 3 sec, msg retries: 1, max concurrency: 10). The workers send batches of messages to the queue and sometimes it gets overloaded thus we expected auto scaling to work, but we've not yet seen auto scaling(concurrency) working there.
For some reason the maximum reference in the metrics graph is 1:
Thanks!

CCharlie Can you help me understand this a bit more? Are multiple Worker scripts a possib...

D

divby0OP•10/19/23, 6:23 AM

Of course : )

So basically I want to locally debug the queue producer and queue consumer. The queue producer is a cloudflare pages app, built on Remix. The queue consumer is a worker with plain typescript, no framework.

I kept the worker that simple intentionally because it doesn't need database access or routing or whatever, it's basically a proxy for queue consumption (because cloudflare pages apps can't consume from a queue yet and other limitations).

If it was possible, I would use only the pages app as the producer and consumer. But my solution with having a separate consumer and producer is really not even what I want, but cloudflare's restriction of me not being able to use pages for everything means I need to split queue production and consumption.

I can't put the queue consumption into the pages app because pages apps can't be consumers. I can't put the queue production into the worker script because I don't have access to database and everything there.

Currently I just have a route on the pages app that I use as a workaround, but not being able to locally test pages<->workers communication via a queue is not nice

G

GuillaumeLakano•10/19/23, 7:41 AM

Hi! In the Known Issues for Local Dev, the documentation explain the producer and consumer need to be on the same worker.
How to do when multiple workers need to be a producer, but we only need one consumer ? After each line changed of our consumer code, we need to copy it to our ~20 workers that are producers ?
I known it's temporary, but it's really not useful to start to works with Queue. Do you have any suggestion to make it more useful please?

GGuillaumeLakano Hi! In the Known Issues for Local Dev, the documentation explain the producer an...

G

GuillaumeLakano•10/19/23, 8:28 AM

From the documentation: « Each queue can only have one consumer Worker connected to it. If you try to connect multiple consumers to the same queue, you will encounter an error when attempting to publish that Worker. »
So, if we have different workers that are producers to the same queue, how can we make only one consumer in local dev ?

CCharlie Hey @GuillaumeLakano great question. No need to copy the consumer to each worker...

G

GuillaumeLakano•10/20/23, 4:44 PM

Thank you for the answer, I'll try, but from what I've understood from documentation, this wasn't possible if your producers aren't in the same worker.
We have around ~20 workers that are producers ( to push in the queue an email notification to send ), but we only want one consumer in a separate worker.

I

iustin.pericica•10/20/23, 5:19 PM

Hi there,
We have a project based on cloudflare workers & queues. We currently have 1 queue with this config ( batch size:50, wait time: 3 sec, msg retries: 1, max concurrency: 10). The workers send batches of messages to the queue and sometimes it gets overloaded thus we expected auto scaling to work, but we've not yet seen auto scaling(concurrency) working there.
For some reason the maximum reference in the metrics graph is 1:
Thanks!

Iiustin.pericica Hi there, We have a project based on cloudflare workers & queues. We currently h...

M

Matt Silverlock•10/21/23, 3:17 PM

What is your configuration? Do you have a high rate of errors? Concurrency backs off (per the docs) when errors increase.

O

Old Man Maker•10/22/23, 7:57 AM

I've got a dead basic consumer and I am getting this message:

Unexpected fields found in queues.consumers[0] field: "services"

Unexpected fields found in queues.consumers[0] field: "services"

Unexpected fields found in queues.consumers[0] field: "services"

Unexpected fields found in queues.consumers[0] field: "services"

I am unsure what this is refering to?

O

Old Man Maker•10/22/23, 7:58 AM

This is the consumer:

async queue(batch: MessageBatch<TransactionModel>, env: Bindings): Promise<void> {
        
        try {
        
            let messages = JSON.stringify(batch.messages);
            console.log(`consumed from our queue: ${JSON.stringify(messages)}`);


        } catch (error) {
            console.log('Error=>',error);
        }
      },

async queue(batch: MessageBatch<TransactionModel>, env: Bindings): Promise<void> {
        
        try {
        
            let messages = JSON.stringify(batch.messages);
            console.log(`consumed from our queue: ${JSON.stringify(messages)}`);


        } catch (error) {
            console.log('Error=>',error);
        }
      },

async queue(batch: MessageBatch<TransactionModel>, env: Bindings): Promise<void> {
        
        try {
        
            let messages = JSON.stringify(batch.messages);
            console.log(`consumed from our queue: ${JSON.stringify(messages)}`);


        } catch (error) {
            console.log('Error=>',error);
        }
      },

async queue(batch: MessageBatch<TransactionModel>, env: Bindings): Promise<void> {
        
        try {
        
            let messages = JSON.stringify(batch.messages);
            console.log(`consumed from our queue: ${JSON.stringify(messages)}`);


        } catch (error) {
            console.log('Error=>',error);
        }
      },

K

kian•10/22/23, 9:19 AM

It's in your Wrangler.toml

Kkian It's in your Wrangler.toml

O

Old Man Maker•10/22/23, 11:41 AM

Yeah - not sure if it should be classified as a bug though? The definition of a service worker anywhere lower in the toml appears to cause these types of error messages. Because mine was below the queue consumer, it looked like it was an error attached to the queue def

Moving it to the top sorted it

K

kian•10/22/23, 11:45 AM

It’s not a bug, just how TOML works

K

kian•10/22/23, 11:45 AM

You can change to using [[services]] with keys below it instead of services = [{…}]

O

Old Man Maker•10/22/23, 11:48 AM

Thanks folks

Original message was deleted

O

Old Man Maker•10/22/23, 11:51 AM

Correct - it was a perfect storm of mess. Queue defined with correct fields and then a heap of other stuff under it commented out, and then the services definition, - so looking at the toml at the point of the queue looked like it was smoking plastic... scolling down, and it turns out I was smoking plastic

Original message was deleted

O

Old Man Maker•10/22/23, 11:53 AM

I have a love hate relationship with it at this point.

I kinda wish that every single entry was standardized, but in this case, the flexibility of it, is also the killer. That and the fact I was rushing and not cleaning up my rubbish - self inflicted pain here for sure

O

Old Man Maker•10/23/23, 10:47 AM

hi folks - I got a worker with a queue consumer, and a binding to a KVNamespace.

I also have this as a HONO app - and I'm not sure if my bindings are actually working at all (is there an easy way to test? they always appear as {} in console.log)

type Bindings = {
    AIS_ALL_TRANSACTIONS:KVNamespace
}


export default {
    fetch: app.fetch,
    async queue(batch: MessageBatch<TransactionModel>, env: Bindings): Promise<void> {

}}

type Bindings = {
    AIS_ALL_TRANSACTIONS:KVNamespace
}


export default {
    fetch: app.fetch,
    async queue(batch: MessageBatch<TransactionModel>, env: Bindings): Promise<void> {

}}

type Bindings = {
    AIS_ALL_TRANSACTIONS:KVNamespace
}


export default {
    fetch: app.fetch,
    async queue(batch: MessageBatch<TransactionModel>, env: Bindings): Promise<void> {

}}

type Bindings = {
    AIS_ALL_TRANSACTIONS:KVNamespace
}


export default {
    fetch: app.fetch,
    async queue(batch: MessageBatch<TransactionModel>, env: Bindings): Promise<void> {

}}

In this case, will the env parameter correctly get the bindings? It does not appear to work for me. I've looked at a few options, but no clearer on a solution.

K

kian•10/23/23, 10:58 AM

Seems fine to me

Kkian Seems fine to me

O

Old Man Maker•10/23/23, 11:27 AM

Yeah, I'm a little stumped. Definitely can't read or write to the kv and the dashboard logging is show me all the console logs up to the try/catch error and then no information from the error, so I'm kinda blind here

K

kian•10/23/23, 11:29 AM

What does your Wrangler.toml look like?

O

Old Man Maker•10/23/23, 11:31 AM

Normal:

[[kv_namespaces]]
binding = "AIS_ALL_TRANSACTIONS"
id = "xxxxx" <=I have the correct Id in here

[[queues.consumers]]
queue = "my-q" <=Correct queue here as well



# Optional: Configure batching and retries: https://developers.cloudflare.com/queues/learning/batching-retries/
# max_batch_size = 10 //The maximum number of messages to deliver before the timeout
# max_batch_timeout = 30 //default is 5 seconds
# max_retries = 10 //default to 3

# all dead transactions go to the following dead letter queue
dead_letter_queue = "transaction-dead-q"

[[kv_namespaces]]
binding = "AIS_ALL_TRANSACTIONS"
id = "xxxxx" <=I have the correct Id in here

[[queues.consumers]]
queue = "my-q" <=Correct queue here as well



# Optional: Configure batching and retries: https://developers.cloudflare.com/queues/learning/batching-retries/
# max_batch_size = 10 //The maximum number of messages to deliver before the timeout
# max_batch_timeout = 30 //default is 5 seconds
# max_retries = 10 //default to 3

# all dead transactions go to the following dead letter queue
dead_letter_queue = "transaction-dead-q"

[[kv_namespaces]]
binding = "AIS_ALL_TRANSACTIONS"
id = "xxxxx" <=I have the correct Id in here

[[queues.consumers]]
queue = "my-q" <=Correct queue here as well



# Optional: Configure batching and retries: https://developers.cloudflare.com/queues/learning/batching-retries/
# max_batch_size = 10 //The maximum number of messages to deliver before the timeout
# max_batch_timeout = 30 //default is 5 seconds
# max_retries = 10 //default to 3

# all dead transactions go to the following dead letter queue
dead_letter_queue = "transaction-dead-q"

[[kv_namespaces]]
binding = "AIS_ALL_TRANSACTIONS"
id = "xxxxx" <=I have the correct Id in here

[[queues.consumers]]
queue = "my-q" <=Correct queue here as well



# Optional: Configure batching and retries: https://developers.cloudflare.com/queues/learning/batching-retries/
# max_batch_size = 10 //The maximum number of messages to deliver before the timeout
# max_batch_timeout = 30 //default is 5 seconds
# max_retries = 10 //default to 3

# all dead transactions go to the following dead letter queue
dead_letter_queue = "transaction-dead-q"

Deploy shows its correctly attached

O

Old Man Maker•10/23/23, 11:34 AM

Is there anyway to get a catch error to show in the dashboard logs?

O

Old Man Maker•10/23/23, 2:50 PM

Moving this to KV channel as it is not queue specific

O

Old Man Maker•10/23/23, 3:31 PM

ok - I now understand what is going wrong.
In example code for queues, we often use a foreach loop to work through the batch. This has a nasty bug (the JS side of it) - I remembered coming across a similar issue a while back with async foreach loops in Node.
So, any async work needing to be done within the scope of the queue, you need to use a plain old for loop. This was hours and hours of debugging. I hope no-one else hits this, or at least people will know.

OOld Man Maker ok - I now understand what is going wrong. In example code for queues, we often ...

U

Unsmart•10/23/23, 3:41 PM

await does nothing on a foreach so this isnt really a bug its expected even if it seems weird: https://stackoverflow.com/questions/37576685/using-async-await-with-a-foreach-loop

U

Unsmart•10/23/23, 3:44 PM

I believe

Error Lens

Error Lens

Error Lens

Error Lens is the extension I have on vs code that makes it a bit easier to see these errors for me at least

OOld Man Maker ok - I now understand what is going wrong. In example code for queues, we often ...

J

John Spurlock•10/23/23, 3:45 PM

you are not the first to encounter this - maybe the cloudflare queue examples should explicitly mention avoiding

.forEach

.forEach

.forEach

.forEach

Debugging fetch in a consumer

UUnsmart I believe `Error Lens` is the extension I have on vs code that makes it a bit ea...

O

Old Man Maker•10/23/23, 4:31 PM

I'll take a look at that extension for sure. Thanks so much

S

Stefan•10/23/23, 10:30 PM

Hi, i'm currently trying to set up a queue between two different workers, one hosts the main application (the producer), and the other is responsible for firing off verification emails (the consumer), but what i'm not clear about and can't find documentation on, is that, assuming I have to have two wrangler.toml files (which my understanding is that i do), then how do i configure the queue to be exposed to both the consumer and the producer if they are in different files?

SStefan Hi, i'm currently trying to set up a queue between two different workers, one ho...

O

Old Man Maker•10/24/23, 7:13 AM

You add the producer details to the toml file of the producer workers (you can have multiple) - and you add the consumer details to the toml of the worker that is going to manage the queue.

S

Stefan•10/24/23, 8:47 AM

Okay so always pair the queue with the consumer, thank you!

�

😈 Donkey 💫•10/25/23, 5:29 AM

Hi everyone,
I have a question:

If:
Client http -> Producer worker -> queue -> consumer worker -> external http service api

How to make producer awaiting, then return response from "external service api" back to client

�

😈 Donkey 💫•10/25/23, 5:29 AM

is that possible?

K

kian•10/25/23, 5:43 AM

You can’t, no

K

kian•10/25/23, 5:43 AM

That wouldn’t be a use-case for Queues but rather Service Bindings

CCharlie What kind of message throughput are you atttempting to send here? What's the err...

A

arbfay•10/25/23, 12:17 PM

I found the problem. The issue was within a DO - I added a

this.env = env

this.env = env

this.env = env

this.env = env in the constructor to be able to call bindings from methods, then tried to send a last message to a queue right after calling

this.state.storage.deleteAll()

this.state.storage.deleteAll()

this.state.storage.deleteAll()

this.state.storage.deleteAll(), but it turns out calling

deleteAll()

deleteAll()

deleteAll()

deleteAll() also deletes the current memory and references of the object (not just storage), so the binding was lost. The error message is a bit confusing though - it made me believe the error was on CF's side (checked status page as well).

B

bret_pat•10/26/23, 10:22 AM

Are queue consumers going to be using the new "standard" worker pricing model in the future? Or at least, have the option to not be unbound?

Bbret_pat Are queue consumers going to be using the new "standard" worker pricing model in...

Y

Yongyi•10/26/23, 11:40 AM

i am also wondering the same thing

https://developers.cloudflare.com/queues/learning/local-development/#known-issues This is a blocker

Similar Threads