@Matt, apologies on keeping @'ing you but I've heard nothing from the support ticket. If there any t

@Matt, apologies on keeping @'ing you but I've heard nothing from the support ticket.
If there any time lines or information you can provide, that would be really helpful. I'll have to start looking at other services to accomodate the Queue, like moving to SQS as it's starting to impact business

Ccg @Matt, apologies on keeping @'ing you but I've heard nothing from the support ti...

Matt Silverlock•11/28/23, 12:32 AM

Unfortunately we’re still investigating. I hate to have this take so long but Queues is still in beta, and so issues like this (which have been rare) may still crop up. I also assume you’ve created a fresh queue and still reproduce this.

cgOP•11/28/23, 12:55 AM

Thanks, I appreciate the reply. Yea, a fresh queue still had the issue.

cgOP•11/28/23, 1:26 AM

Could I confirm that when calling

QUEUE.send()

QUEUE.send()

within a consumer, that it doesn't spawn a new worker within the same "thread"

cgOP•11/28/23, 1:26 AM

I'm seeing

Queue sendBatch failed: Subrequest depth limit exceeded.  This request recursed through Workers too many times.

Queue sendBatch failed: Subrequest depth limit exceeded.  This request recursed through Workers too many times.

but the message that has invoked the consumer should not reach this limit at all

cgOP•11/28/23, 1:28 AM

This is the code that is being hit (has been modified)

cgOP•11/28/23, 1:29 AM

export const handleMessage = async ({
  message,
  env,
}) => {
  const { items, nextPageToken } =
    await api.listByStatus({
      limit: 100,
      nextPageToken: message.nextPageToken,
      status: 'approved',
    });

  if (nextPageToken) {
    await sendQueueMessage({
      message: {
        type: '<REDACTED>',
        nextPageToken,
      },
      env,
    });
  }

  const toBeProcessed = items
    .filter((f) => !f.processed)

  await sendBatchQueueMessage({
    batch: toBeProcessed.map((transaction) => ({
      message: {
        transaction,
        type: '<OTHERMESSAGE>',
      },
    })),
    env,
  });
};

export const handleMessage = async ({
  message,
  env,
}) => {
  const { items, nextPageToken } =
    await api.listByStatus({
      limit: 100,
      nextPageToken: message.nextPageToken,
      status: 'approved',
    });

  if (nextPageToken) {
    await sendQueueMessage({
      message: {
        type: '<REDACTED>',
        nextPageToken,
      },
      env,
    });
  }

  const toBeProcessed = items
    .filter((f) => !f.processed)

  await sendBatchQueueMessage({
    batch: toBeProcessed.map((transaction) => ({
      message: {
        transaction,
        type: '<OTHERMESSAGE>',
      },
    })),
    env,
  });
};

cgOP•11/28/23, 1:31 AM

These are the two helper functions:

export const sendQueueMessage = ({ message, env }: SendQueueMessage) => {
  return env.PROCESS_QUEUE.send(message, { contentType: 'json' });
};

export const sendBatchQueueMessage = ({
  batch,
  env,
}: SendBatchQueueMessage) => {
  const ch = chunk(batch, 20);
  return Promise.all(
    ch.map((c) => {
      return env.PROCESS_QUEUE.sendBatch(
        c.map((body) => ({ body: body.message, contentType: 'json' })),
      );
    }),
  );
};

export const sendQueueMessage = ({ message, env }: SendQueueMessage) => {
  return env.PROCESS_QUEUE.send(message, { contentType: 'json' });
};

export const sendBatchQueueMessage = ({
  batch,
  env,
}: SendBatchQueueMessage) => {
  const ch = chunk(batch, 20);
  return Promise.all(
    ch.map((c) => {
      return env.PROCESS_QUEUE.sendBatch(
        c.map((body) => ({ body: body.message, contentType: 'json' })),
      );
    }),
  );
};

kian•11/28/23, 2:02 AM

Concurrent consumers are in the same Worker 'instance' I believe, so you can sometimes run into limits that you wouldn't expect to hit.

Unsmart•11/28/23, 4:26 AM

I'm guessing durable object alarms dont have this subrequest depth tracking? I think quite a few people use it for crons so could use that for the infinite loop things

cgOP•11/28/23, 6:07 AM

Oh interesting - I'm not really sure how to deal with this though. I could try setting this:

await api.listByStatus({
      limit: 100,

await api.listByStatus({
      limit: 100,

to something like limit 10 or limit 5

cgOP•11/28/23, 6:08 AM

But from looking at the code in it's current state, I'd only expect like 6 requests (4 batches of 20 + maybe two others)

cgOP•11/28/23, 6:08 AM

Which feels happily within the 1000 request limit

Kkian Concurrent consumers are in the same Worker 'instance' I believe, so you can som...

cgOP•11/28/23, 6:12 AM

Oh, well, I must say that limitation kinda sucks - trying to juggle the number of requests a worker can make is tricky enough without also having to consider the possibility of this worker being run concurrently and accounting for those workers. Especially considering that the

max_concurrent

max_concurrent

is default

auto

auto

Ccg Which feels happily within the 1000 request limit

kian•11/28/23, 6:12 AM

Depth is a different limit than exceeding the 50/1000 subrequests limits - it's not really documented and is generally only caused by recursion.

cgOP•11/28/23, 6:13 AM

I've had to set my max_batch to be

because I've already come into trouble with too many messages being handled by 1 workers and hitting subrequest limits

Kkian Depth is a different limit than exceeding the 50/1000 subrequests limits - it's ...

cgOP•11/28/23, 6:15 AM

Right, well, I guess I'm a bit lost as to what to do. We will have to see what @AlexRobinson comes up with tomorrow

cgOP•11/28/23, 6:19 AM

Am I using queues incorrectly?
Trying to paginate through data from an API felt like Queues were a good fit

Unsmart•11/28/23, 6:46 AM

Durable objects + alarms would be better for that imo

cgOP•11/28/23, 6:49 AM

mmm, I don't really need to global-ness that DO offers

cgOP•11/28/23, 6:50 AM

but I'll explore it, ty

Ccg Am I using queues incorrectly? Trying to paginate through data from an API felt...

Matt Silverlock•11/28/23, 1:47 PM

This is a sharp edge we should just fix.

Shihmin Lee•11/28/23, 9:35 PM

Hey guys. I've also created an issue about the queue not logging, processing jobs, or stopping unexpectedly without generating logs. Would anyone have insight into what might be causing this behavior? https://github.com/cloudflare/workers-sdk/issues/4517

GitHub

🐛 BUG: Queue are not working correctly when you have 2 consumers · ...

Which Cloudflare product(s) does this pertain to? Queues What version(s) of the tool(s) are you using? 3.17.1 What version of Node are you using? 20.10.0 What operating system are you using? Mac De...

Shihmin Lee•11/29/23, 3:25 AM

I have another question. When I am consuming the queue messages, I notice that sometimes I find myself in a scenario where both the queue and kv are failing. These are the logs I receive through Logpush / Datadog. Do you have any suggestions on how this could happen and how do I get around it?

kian•11/29/23, 9:16 AM

Sounds like the same issue as the conversation above.

SShihmin Lee I have another question. When I am consuming the queue messages, I notice that s...

Matt Silverlock•11/29/23, 12:28 PM

How long have you been using Queues? Trying to understand if you just set it up and saw this, or if you saw this after using Queues for some time.

drk•11/29/23, 1:18 PM

are there good examples of a pages/workers/queue monorepo? i deploy a remix project to pages currently per cloudflare remix guides and want to leverage queues. so far it seems like a single queue is the only documented setup and given remix+pages constraints that makes sense, but i'm curious about additional workers as consumers and my remix app as a producer primarily.

drk•11/29/23, 1:19 PM

and specifically running multiple workers/consumers locally so i can develop end-to-end, but i think i read that you can only run a single queue consumer at a time locally?

drk•11/29/23, 1:20 PM

also looking for tips on general dev workflow for a setup like this. like if it's best to work on consumers in isolation, deploy those (or preview deploy them for dev purposes) and have my remix app running locally use a dev specific queue that's live in cloudflare.

MMatt Silverlock How long have you been using Queues? Trying to understand if you just set it up ...

Shihmin Lee•11/29/23, 1:53 PM

I am just using Queues for a few days. I suspect this is me hitting some rate limits? I also notice a similar issue for Queue + D1.

Shihmin Lee•11/29/23, 1:59 PM

Although I doubt that I was hitting the rate limits since I am not seeing 429, also I only have 2 workers on 2 queue with 1 consumer each inserted like 1000 records over 5 mins on avg.

SShihmin Lee I am just using Queues for a few days. I suspect this is me hitting some rate li...

Matt Silverlock•11/29/23, 2:01 PM

Your issue appears identical to queues

Shihmin Lee•11/29/23, 2:01 PM

Gotcha. I've never seen Subrequest depth limit error message though.

SShihmin Lee Gotcha. I've never seen Subrequest depth limit error message though.

Matt Silverlock•11/29/23, 4:23 PM

Per that chat message, that error does not always manifest.

TTyler Benfield Is there any known degradation to queues in the last ~12 hours? We are seeing so...

Matt Silverlock•11/30/23, 1:14 PM

Not specific to the last 12 hours, but we have had a couple of reports about consumers not being invoked. We’re looking into the root cause.

TTyler Benfield Thanks @Matt! Let us know if there's any information we can provide. We're going...

Matt Silverlock•11/30/23, 1:36 PM

Note that it’s not related to concurrency - that just makes it more obvious.

If you have an account ID and queue ID we can add to our list.

MMatt Silverlock Per that chat message, that error does not always manifest.

Shihmin Lee•11/30/23, 3:33 PM

Hey Matt. So do you know if there's a timeline that this is going to be addressed? I'm interested in moving my infra from S3 to using R2, D1, Queues and KVs and such. But so far it's been harsh to see that I am getting so many 503s for combinations of kv, queue, d1, ... etc in the queue msg as well as all the immediate messages that got consumed.

SShihmin Lee Hey Matt. So do you know if there's a timeline that this is going to be addresse...

Matt Silverlock•11/30/23, 3:38 PM

We're working on it, but can't give an accurate timeline as it's non-trivial. Days to weeks, not months. Before Queues hits GA.

Shihmin Lee•11/30/23, 3:46 PM

Cool. That would be so nice.

Dj•11/30/23, 5:06 PM

Queues are awesome btw, I just found out about this product sometime last week. Using it to track some very dumb metrics from a Worker and it was very easy to figure out, both consumers and producers.

Dj•11/30/23, 5:07 PM

kian•11/30/23, 8:21 PM

Serializable object

export const handleMessage = async ({ message, env, }) => { const { items, nextPageToken } = await api.listByStatus({ limit: 100, nextPageToken: message.nextPageToken, status: 'approved', }); if (nextPageToken) { await sendQueueMessage({ message: { type: '<REDACTED>', nextPageToken, }, env, }); } const toBeProcessed = items .filter((f) => !f.processed) await sendBatchQueueMessage({ batch: toBeProcessed.map((transaction) => ({ message: { transaction, type: '<OTHERMESSAGE>', }, })), env, }); };

export const sendQueueMessage = ({ message, env }: SendQueueMessage) => { return env.PROCESS_QUEUE.send(message, { contentType: 'json' }); }; export const sendBatchQueueMessage = ({ batch, env, }: SendBatchQueueMessage) => { const ch = chunk(batch, 20); return Promise.all( ch.map((c) => { return env.PROCESS_QUEUE.sendBatch( c.map((body) => ({ body: body.message, contentType: 'json' })), ); }), ); };

@Matt, apologies on keeping @'ing you but I've heard nothing from the support ticket. If there any t

Similar Threads

Similar Threads

Similar Threads