Hello, I'm having a problem regarding queues. I have a worker that streams and parses a big CSV file

Hello, I'm having a problem regarding queues. I have a worker that streams and parses a big CSV file and puts one JSON object per row in a batch of 100 to a Queue. Locally, on dev, the flow works flawlessly, I can parse and enqueue a 250Mb CSV file with no problem and using low heap space. But in cloudflare, for some unknown reason the worker stops enqueuing at around 2600 or so objects with no error or warning whatsoever.

The code of the worker looks like this:

export async fetch(request: Request, env: Environment, ctx: ExecutionContext) {
  const response = await fetch(someUrl);
  const objectsStream = response.body!.getReader()
    .pipeThrough(new TextDecoderStream())
    .pipeThrough(new CSVParserStream());
  const enqueue = async () => {
    for await (const objects of take(100, objectsStream)) {
      await env.MY_QUEUE.sendBatch(objects.map(object => ({
        body: object,
        contentType: "json",
      })));
    }
  };
  ctx.waitUntil(
    enqueue()
      .then(() => {
        console.log("Success");
      })
      .catch(console.error)
  );
  return new Response("OK");
}

export async fetch(request: Request, env: Environment, ctx: ExecutionContext) {
  const response = await fetch(someUrl);
  const objectsStream = response.body!.getReader()
    .pipeThrough(new TextDecoderStream())
    .pipeThrough(new CSVParserStream());
  const enqueue = async () => {
    for await (const objects of take(100, objectsStream)) {
      await env.MY_QUEUE.sendBatch(objects.map(object => ({
        body: object,
        contentType: "json",
      })));
    }
  };
  ctx.waitUntil(
    enqueue()
      .then(() => {
        console.log("Success");
      })
      .catch(console.error)
  );
  return new Response("OK");
}

export async fetch(request: Request, env: Environment, ctx: ExecutionContext) {
  const response = await fetch(someUrl);
  const objectsStream = response.body!.getReader()
    .pipeThrough(new TextDecoderStream())
    .pipeThrough(new CSVParserStream());
  const enqueue = async () => {
    for await (const objects of take(100, objectsStream)) {
      await env.MY_QUEUE.sendBatch(objects.map(object => ({
        body: object,
        contentType: "json",
      })));
    }
  };
  ctx.waitUntil(
    enqueue()
      .then(() => {
        console.log("Success");
      })
      .catch(console.error)
  );
  return new Response("OK");
}

export async fetch(request: Request, env: Environment, ctx: ExecutionContext) {
  const response = await fetch(someUrl);
  const objectsStream = response.body!.getReader()
    .pipeThrough(new TextDecoderStream())
    .pipeThrough(new CSVParserStream());
  const enqueue = async () => {
    for await (const objects of take(100, objectsStream)) {
      await env.MY_QUEUE.sendBatch(objects.map(object => ({
        body: object,
        contentType: "json",
      })));
    }
  };
  ctx.waitUntil(
    enqueue()
      .then(() => {
        console.log("Success");
      })
      .catch(console.error)
  );
  return new Response("OK");
}

kian•11/2/23, 5:55 AM

Is that 2600 objects taking 30 seconds by any chance?

Kkian Is that 2600 objects taking 30 seconds by any chance?

paper___OP•11/2/23, 2:24 PM

Not really, it is 5 seconds top

Ppaper___Hello, I'm having a problem regarding queues. I have a worker that streams and p...

robmarscher•11/3/23, 4:31 PM

I had a similar issue that I think was due to the 1000 max subrequests per worker invocation. I think the queue send counts as a subrequest. https://developers.cloudflare.com/workers/platform/limits/#how-many-subrequests-can-i-make

That doesn't add up with 2600 objects though... that would only be 26 requests with a batch size of 100. 128 MB is the memory limit... but I assume that wouldn't be hit when you are streaming the body.

With sendBatch, there is a 256KB max size... maybe you are hitting that at a certain point in your data set and reducing the number of objects in sendBatch would fix?

A batch can contain up to 100 messages, though items are limited to 128 KB each, and the total size of the array cannot exceed 256 KB.

https://developers.cloudflare.com/queues/platform/javascript-apis/#queue

For the subrequest limit, I solved it by storing the uploaded csv on R2 and then using another queue to read the csv. That queue handler would track how many rows it processed and then once it reached 750 (arbitrary number to leave buffer under the 1000 limit) it would enqueue another task to read the csv - passing the number of rows to skip and the path to the csv on R2 in the queue message data. I also used a Durable Object to track the progress and push updates via websocket to the client that uploaded the csv.

Ppaper___Hello, I'm having a problem regarding queues. I have a worker that streams and p...

Unsmart•11/3/23, 4:35 PM

Try running the code without using ctx.waitUntil and see if you get any errors... its possible that waitUntil is swallowing the error

UUnsmart Try running the code without using ctx.waitUntil and see if you get any errors.....

Matt Silverlock•11/3/23, 7:54 PM

This is very common - you will swalllow the exception inside

waitUntil

waitUntil

unless you are actively logging

cg•11/3/23, 9:02 PM

Are Queues down at the moment?

Ccg Are Queues down at the moment?

Matt Silverlock•11/4/23, 11:36 AM

We observed a small blip affecting some Queues around the time of your post.

Rrobmarscher I had a similar issue that I think was due to the 1000 max subrequests per worke...

paper___OP•11/6/23, 12:42 PM

Thanks Rob! I'm going to try checking the size of the batch and see if I'm hitting the limit. When you say 128kb each message, that is JSON encoded right?

UUnsmart Try running the code without using ctx.waitUntil and see if you get any errors.....

paper___OP•11/6/23, 1:19 PM

 [ERROR]   Error: Worker exceeded memory limit.

 [ERROR]   Error: Worker exceeded memory limit.

paper___OP•11/6/23, 1:20 PM

The only thing I'm thinking might be the problem is a mem leak in the csv parsing function

Ppaper___The only thing I'm thinking might be the problem is a mem leak in the csv parsin...

Matt Silverlock•11/6/23, 1:36 PM

If your decoder isn’t streaming correctly you will definitely run out of memory.

MMatt Silverlock If your decoder isn’t streaming correctly you will definitely run out of memory.

paper___OP•11/6/23, 3:26 PM

I'm using

csv-parse

csv-parse

, maybe the internal Buffer is not flushing correctly.

paper___OP•11/6/23, 3:26 PM

I'm creating a snapshot right now to check that

Nnil Hello everyone 🙂 I am having an issue with concurrency, for some reason my que...

Matt Silverlock•11/7/23, 5:51 PM

What is your account ID, and what are the settings for
max_batch_size
max_batch_size
and
max_batch_timeout
max_batch_timeout
on the queue? (I see them now, missed those).

What is your message volume during that period? 567 messages in a 24 hour period isn't a lot, and so your consumer shouldn't have to scale up. If your consumer is making calls to an external service before ack'ing that will also increase the measured delay.

Nnil Thanks Matt! This is my account ID: `34c0d8412965b33f782ad043d255542c` the queue...

Matt Silverlock•11/7/23, 5:58 PM

Can you share your message rate chart as well, for the same time period?

Nnil also regarding what you said, indeed before ack'ing the messages multiple calls ...

Matt Silverlock•11/7/23, 6:06 PM

No: batch timeout is on the producer side. It does not determine how the consumer scales. Your volume/message rate is very low, and (right now) we prefer to avoid scaling too quickly on the consumer side.

jasonkuhrt•11/7/23, 6:29 PM

Question about queues I guess I should have put here? workers-and-pages-helpAre queue producers and consumers supposed to work in local development with wrangler?

AKStriker98•11/8/23, 5:18 PM

Question about the queues beta. Given the public beta has been out for almost a year now, is there a rough timeline or progress level for how far along the public-beta is for full production? We would love to use queues for logging tasks but don’t yet feel comfortable using a beta product.

AAKStriker98 Question about the queues beta. Given the public beta has been out for almost a ...

Matt Silverlock•11/8/23, 8:19 PM

We want to get Queues into GA by end of Q1 next year. There’s architectural work to get per-queue throughput into the thousands-per-queue that’s happening behind the scenes.

lenqbit•11/9/23, 9:12 AM

Hello, what is the quickest/fastest way to purge a queue? We are in a situation where our queue has accumulated too much backlog that our consumer can not cope with it anymore.
We would like to purge the whole queue first as way to mitigate the issue before we look at finding out the underlying issue and work on a long term fix.
Is it a case of deploying a new version of our consumer that just acks the batches with say

ackAll()

ackAll()

?
Or is there an API that can be invoked from outside a worker?

lenqbit•11/9/23, 9:13 AM

I've seen that there is a way to list and ack message in the dashboard but this will not help as it limits the batch size to 100

Llenqbit Hello, what is the quickest/fastest way to purge a queue? We are in a situation ...

Matt Silverlock•11/9/23, 11:20 AM

There’s not a “purge all” method - re-deploying a consumer that acks every batch and returns immediately is the fastest way right now. Or deleting the queue/rebinding to a new queue.

rikka0w0•11/9/23, 1:01 PM

According to the JavaScript API documentation of Queue, inside the 'export default' block, the queue consumer is a separate handler(queue). Is there a way to poll a queue inside a running fetch()? Or it is possible to get notified inside a running fetch function for changes of a queue? Im looking for a way to implement communication between two running worker instances.

kian•11/9/23, 1:02 PM

Not without using something like Durable Objects where the Workers can read/write state for the other

Kkian Not without using something like Durable Objects where the Workers can read/writ...

rikka0w0•11/9/23, 2:32 PM

Will it be possible to implement such function with kv?

kian•11/9/23, 2:43 PM

It’s not ideal because KV is eventually consistent and reads are cached for up to 60 seconds, so polling doesn’t really work

kian•11/9/23, 2:44 PM

If you polled, and there was nothing, it’d still read as nothing even if your queue wrote to KV for up to 60 seconds.

Stockton•11/9/23, 4:55 PM

Are Queues right for me?

I'm looking to send many events to a queue, wait 30 seconds, and then send a single array with all accumulated events to a destination.

When I removed the batch.messages for each part so that it wouldn't process each event individually, it failed.

SStockton Are Queues right for me? I'm looking to send many events to a queue, wait 30 s...

Matt Silverlock•11/9/23, 6:11 PM

Can you expand? You control batching via your queue settings (https://developers.cloudflare.com/queues/learning/batching-retries/) - the messages in the consumer are always accessed via

batch.messages

batch.messages

, whether it's a batch or 1 or 100.

Batching and Retries · Cloudflare Queues

When configuring a consumer Worker for a queue, you can also define how messages are batched as they are delivered.

MMatt Silverlock Can you expand? You control batching via your queue settings (https://developers...

Stockton•11/9/23, 6:44 PM

Thank you so much for replying. Essentially what's happened is that consumer did send the batch of events as a single array to the destination, but it thought that it failed. So it would retry (sending the same batch over and over) until it reached the retry limit.

SStockton Thank you so much for replying. Essentially what's happened is that consumer did...

Matt Silverlock•11/9/23, 6:47 PM

It would "fail" if the

queue

queue

handler throws an exception, otherwise returning without an error is considered success.

Matt Silverlock•11/9/23, 6:47 PM

Alternatively you can explicitly ack or retry messages: https://developers.cloudflare.com/queues/learning/batching-retries/#explicit-acknowledgement

Batching and Retries · Cloudflare Queues

When configuring a consumer Worker for a queue, you can also define how messages are batched as they are delivered.

MMatt Silverlock Alternatively you can explicitly ack or retry messages: https://developers.cloud...

Stockton•11/9/23, 6:51 PM

The batch was definitely delivered (as a single batch) because I'm not interating over batch.messages. So would I still need to explicitely ack every individual message even though I'm sending them as a single batch?

SStockton The batch was definitely delivered (as a single batch) because I'm not interatin...

Matt Silverlock•11/9/23, 7:33 PM

You can

ackAll()

ackAll()

as described there too

Stockton•11/9/23, 8:53 PM

Ahh got it. Wasn't working in dev. Thank you so much for the help.

Ashley•11/10/23, 2:14 PM

Random Q, is there a reason https://developers.cloudflare.com/workers/wrangler/commands/ doesn't have queues on it? (was looking for the commands and always go here, but can't see it on there, which is odd as other products in beta are on there