If the worker errors it wont delete the messages from the queue so they would get retried eventually

UUnsmart If the worker errors it wont delete the messages from the queue so they would ge...

A

Adi//•9/28/23, 5:41 AM

oh this is super cool then imo, trying to understand how this works,

are messages deleted only after a task is completed? or thats upto me

A

Adi//•9/28/23, 5:41 AM

Also I assume duplicacy is handled directly by cloudflare (one message = one worker instance?)

A

Adi//•9/28/23, 5:43 AM

can someone help me understand these 2?

Screenshot_2023-09-28_at_11.13.36_AM.png

Screenshot_2023-09-28_at_11.13.47_AM.png

CCharlie @Adi// messages are deleted in 3 different circumstances - When the message is e...

A

Adi//•9/28/23, 6:45 PM

Got it understood, this helps

A

Adi//•9/28/23, 6:46 PM

Also I assume that means workers have no runtime limits (wall time) until I am well under cpu limits?

G

gideonred•9/29/23, 2:48 PM

Hello, ex-Cloudflarian here

. Two questions:

Is there a timeline for Queues to be supported in Wrangler dev preview mode?
We currently have Queues enabled for our CF prod account, but not non-prod. Our infrastructure team has reached out, but I haven't heard back from my internal team. Any chance someone here is motived to help me get it enabled?

G

gideonred•9/29/23, 2:58 PM

Also, is there a direct HTTP POST I can do to a queue for testing purposes?

M

Matt Silverlock•9/29/23, 5:46 PM

Gideon!

See this note: https://developers.cloudflare.com/queues/learning/local-development/#known-issues
- we’re working on a fix for this but won’t be this month.
Email me your account ID. Queues requires a paid Workers plan - that’s it. What are you seeing that isn’t working if that is the case?
HTTP POST - there is a dash debugging UI for testing. Not yet a REST API you can use.

Local Development · Cloudflare Queues

Queues support local development workflows using Wrangler, the command-line interface for Workers. Wrangler runs the same version of Queues as …

G

gideonred•9/29/23, 8:19 PM

@Matt !

Thanks for 1 and 3.

For 2: we finally got a reply from our Cloudflare TAM and yes, they stated we need to add a payment method on our non-prod account. I'm working on us setting that up.

M

MetalGear•10/2/23, 3:42 PM

Hey guys,

really loving my solution using only cloudflare services for my backend api but im currently suffering from one problem I am hoping someone can guide me. I am using cloudflare queues to manage events that need to be processed say sending a command to a iot device. That iot device 3rd party api has been giving me issues sometimes where its unnavailable for 3 hours due to maintnanace on their end. So when my worker consumes that queue I need a way to keep retrying all the queue items every hour. until they succeed and then I can mark them as consumed.

M

MetalGear•10/2/23, 4:04 PM

Do you guys think i should just use upstash for now if I need those features?

S

switz•10/3/23, 8:07 PM

whats the best way to retry a message in the queue but on a delay?

S

switz•10/3/23, 8:08 PM

will retries happen immediately?

Sswitz whats the best way to retry a message in the queue but on a delay?

M

MetalGear•10/3/23, 10:22 PM

No good way to do that right now unfortunetely based on my limited experience. I needed a solution fast so i just temporarily moved to qstash from upstash

S

switz•10/3/23, 10:46 PM

alternatively, would be nice to consume a queue from an external application. that would solve the problem for me

S

switz•10/3/23, 10:46 PM

so I only consume items when my server is up and running

S

switz•10/3/23, 10:47 PM

maybe this: https://idearium.github.io/idearium-lib/docs/cloudflare-queues

@idearium/cloudflare-queues | Idearium Lib

Safely process Cloudflare Queues batches and messages.

Sswitz so I only consume items when my server is up and running

M

Matt Silverlock•10/4/23, 12:39 AM

This is coming (HTTP Pull) in the future.

S

switz•10/4/23, 12:40 AM

Sweet, I’d really love that.

S

Sameer Ali•10/4/23, 12:41 AM

I am looking at queue api docs, there is no way to send something to queue outside of the worker. Am i missing anything?

SSameer Ali I am looking at queue api docs, there is no way to send something to queue outsi...

M

Matt Silverlock•10/4/23, 12:46 AM

You would need to write a Worker: https://developers.cloudflare.com/queues/get-started/#write-your-producer-worker

Get started · Cloudflare Queues

Cloudflare Queues is a flexible messaging queue that allows you to queue messages for asynchronous processing. By following this guide, you will …

S

Sameer Ali•10/4/23, 12:47 AM

Thanks

�

😈 Donkey 💫•10/4/23, 5:09 AM

my producer is simple:

export interface Env {
    MY_QUEUE: Queue;
}

export default {
    async fetch(req: Request, env: Env, ctx: ExecutionContext): Promise<Response> {

        // if req.method is not POST , reject
        if (req.method !== 'POST') {
            return new Response('Sorry', { status: 405 });
        }
        const body = await req.json();
        console.log(`got body: ${JSON.stringify(body)}`)
        const result = await env.MY_QUEUE.send({
            body: body,
            contentType: "json",
        });
        console.log(`sent message to queue: ${JSON.stringify(result)}`);
        return new Response(JSON.stringify({
            return_code: "1",
            return_message: "ok"
        }), {
            headers: { 'Content-Type': 'application/json' },
        });
    },
};

export interface Env {
    MY_QUEUE: Queue;
}

export default {
    async fetch(req: Request, env: Env, ctx: ExecutionContext): Promise<Response> {

        // if req.method is not POST , reject
        if (req.method !== 'POST') {
            return new Response('Sorry', { status: 405 });
        }
        const body = await req.json();
        console.log(`got body: ${JSON.stringify(body)}`)
        const result = await env.MY_QUEUE.send({
            body: body,
            contentType: "json",
        });
        console.log(`sent message to queue: ${JSON.stringify(result)}`);
        return new Response(JSON.stringify({
            return_code: "1",
            return_message: "ok"
        }), {
            headers: { 'Content-Type': 'application/json' },
        });
    },
};

export interface Env {
    MY_QUEUE: Queue;
}

export default {
    async fetch(req: Request, env: Env, ctx: ExecutionContext): Promise<Response> {

        // if req.method is not POST , reject
        if (req.method !== 'POST') {
            return new Response('Sorry', { status: 405 });
        }
        const body = await req.json();
        console.log(`got body: ${JSON.stringify(body)}`)
        const result = await env.MY_QUEUE.send({
            body: body,
            contentType: "json",
        });
        console.log(`sent message to queue: ${JSON.stringify(result)}`);
        return new Response(JSON.stringify({
            return_code: "1",
            return_message: "ok"
        }), {
            headers: { 'Content-Type': 'application/json' },
        });
    },
};

export interface Env {
    MY_QUEUE: Queue;
}

export default {
    async fetch(req: Request, env: Env, ctx: ExecutionContext): Promise<Response> {

        // if req.method is not POST , reject
        if (req.method !== 'POST') {
            return new Response('Sorry', { status: 405 });
        }
        const body = await req.json();
        console.log(`got body: ${JSON.stringify(body)}`)
        const result = await env.MY_QUEUE.send({
            body: body,
            contentType: "json",
        });
        console.log(`sent message to queue: ${JSON.stringify(result)}`);
        return new Response(JSON.stringify({
            return_code: "1",
            return_message: "ok"
        }), {
            headers: { 'Content-Type': 'application/json' },
        });
    },
};

after make 1 request via PostMan

there's a lots triggering this function, make massive queues produced to the Queue

any of them, has same logs like below:

"logs": [
    {
      "message": [
        "got body: {\"data\":\"dat hart fd3 3\"}"
      ],
      "level": "log",
      "timestamp": 1696396087813
    },
    {
      "message": [
        "sent message to queue: undefined"
      ],
      "level": "log",
      "timestamp": 1696396087867
    }
  ]

"logs": [
    {
      "message": [
        "got body: {\"data\":\"dat hart fd3 3\"}"
      ],
      "level": "log",
      "timestamp": 1696396087813
    },
    {
      "message": [
        "sent message to queue: undefined"
      ],
      "level": "log",
      "timestamp": 1696396087867
    }
  ]

"logs": [
    {
      "message": [
        "got body: {\"data\":\"dat hart fd3 3\"}"
      ],
      "level": "log",
      "timestamp": 1696396087813
    },
    {
      "message": [
        "sent message to queue: undefined"
      ],
      "level": "log",
      "timestamp": 1696396087867
    }
  ]

"logs": [
    {
      "message": [
        "got body: {\"data\":\"dat hart fd3 3\"}"
      ],
      "level": "log",
      "timestamp": 1696396087813
    },
    {
      "message": [
        "sent message to queue: undefined"
      ],
      "level": "log",
      "timestamp": 1696396087867
    }
  ]

what's wrong with my code? or is this a bug? any helps?

�

😈 Donkey 💫•10/4/23, 6:07 AM

found out
it's because the routes endwith wildcard (star), without slashes rightbefore

( domain.com/a* => bug massive trigger)
( domain.com/a/* => works now)

M

marina.lipeck•10/4/23, 10:22 AM

hey guys! is there a way to increase the limit of queues? We used the request form (https://docs.google.com/forms/d/e/1FAIpQLSd_fwAVOboH9SlutMonzbhCxuuuOmiU1L_I5O2CFbXf_XXMRg/viewform) from the docs a couple of weeks ago, but never got any answer

Google Docs

Cloudflare Developer Platform: Limit Increase Request Form

Want to increase the limits of your current plan? Let us know what your requirements are here.

Mmarina.lipeck hey guys! is there a way to increase the limit of queues? We used the request fo...

M

Matt Silverlock•10/4/23, 11:07 AM

Which limit are you looking to increase?

MMatt Silverlock Which limit are you looking to increase?

M

marina.lipeck•10/4/23, 11:23 AM

this one https://developers.cloudflare.com/queues/platform/limits/#limits:~:text=Queues-,100%20per%20account,-1

Limits · Cloudflare Queues

Many of these limits will increase during Queues’ public beta period. Follow our changelog or join the #queues-beta channel in our Developer Discord …

Mmarina.lipeck this one https://developers.cloudflare.com/queues/platform/limits/#limits:~:text...

M

Matt Silverlock•10/4/23, 11:35 AM

I can't see which request is yours: can you share your account ID here? I'll increase it to 10,000.

MMatt Silverlock I can't see which request is yours: can you share your account ID here? I'll inc...

M

marina.lipeck•10/4/23, 11:37 AM

thanks! I just received the email that increased is done

Mmarina.lipeck thanks! I just received the email that increased is done

M

Matt Silverlock•10/4/23, 11:40 AM

Great! There were two outstanding asks, so was hoping one was yours

S

switz•10/4/23, 2:12 PM

it says you can set it from 0-60, but if you set it to 0 it defaults back to 5

S

switz•10/4/23, 2:12 PM

is 0 possible?

S

switz•10/4/23, 2:13 PM

no delay for batching

S

switz•10/4/23, 2:14 PM

1 second is fine for my use-cases, but in an ideal world things would process as they come in – perhaps as low as 50ms or 200ms if a batch is necessary?

S

switz•10/4/23, 2:15 PM

but yeah, processing 1 at a time is also ok by me, if it means its faster

Original message was deleted

N

nclevenger•10/4/23, 3:44 PM

I just realized I was seeing this behavior too ... and yes, in our case, there are certain things that are very latency sensitive so at low volumes, immediate is best, but as volumes scale then they will naturally batch ... but yes, 50ms or 100ms would be ideal for those ... but we had been trying to set to 0

V

vikiival | koda.art•10/5/23, 10:24 AM

Hello,
has anyone tried honojs or other framework for queue worker?

K

kian•10/5/23, 10:25 AM

Hono is purely for the

fetch

fetch

handler so if you mean you want

fetch

fetch

w/ Hono and

queue

queue

queue

queue handlers in the same project, then just do it like this

export default {
  fetch: app.fetch,
  async queue(...) { ... }
}

export default {
  fetch: app.fetch,
  async queue(...) { ... }
}

export default {
  fetch: app.fetch,
  async queue(...) { ... }
}

export default {
  fetch: app.fetch,
  async queue(...) { ... }
}

Kkian Hono is purely for the `fetch` handler so if you mean you want `fetch` w/ Hono a...

V

vikiival | koda.art•10/5/23, 10:26 AM

Thank you very much ferrisBongoHyper

Kkian Hono is purely for the `fetch` handler so if you mean you want `fetch` w/ Hono a...

V

vikiival | koda.art•10/5/23, 10:39 AM

Offtopic but same also works for

scheduled

scheduled

scheduled

scheduled worker right?

A

Adi//•10/6/23, 6:00 AM

Hi just signed up for the paid worker plan
Hi what does this exactly mean in limits?
I have a queue with batching 1 message,1 second
Does that mean my average requests per second can be max 10?

This is the number of "starts" per second right? Not the number of concurrent queues?

Screenshot_2023-10-06_at_11.28.06_AM.png

AAdi//Hi just signed up for the paid worker plan Hi what does this exactly mean in li...

M

Matt Silverlock•10/6/23, 9:07 AM

It is linked directly to https://developers.cloudflare.com/queues/learning/consumer-concurrency/#enable-concurrency

Consumer concurrency · Cloudflare Queues

Consumer concurrency allows a consumer Worker processing messages from a queue to automatically scale out horizontally to keep up with the rate that …

M

Matt Silverlock•10/6/23, 4:22 PM

If you're using Queues, you can now create up to 10,000 queues per account: https://developers.cloudflare.com/queues/changelog/#more-queues-per-account---up-to-10000

Changelog · Cloudflare Queues

Queue consumers can now scale to 20 concurrent invocations (per queue), up from 10. This allows you to scale out and process higher throughput queues …

U

UnsmartOP•10/6/23, 4:24 PM

Oh nice 20 concurrent invokes also yesterday / 2 days ago Pog

U

UnsmartOP•10/6/23, 4:26 PM

What timezone are the changelogs in for it to says the queue count is the 7th? Or was that just the planned date and wasnt changed when it happened a day early

UUnsmart What timezone are the changelogs in for it to says the queue count is the 7th? O...

M

Matt Silverlock•10/6/23, 5:09 PM

Day early

UUnsmart Oh nice 20 concurrent invokes also yesterday / 2 days ago <:Pog:9027562436916839...

M

Matt Silverlock•10/6/23, 5:10 PM

And yep: queues will auto-scale to 20 concurrent consumers now. This will continue to get higher, but there’s more work to do here, and I’d like to get message delays in place to go alongside so there’s more control over throughput.

F

forslunds•10/6/23, 6:46 PM

Sweet! 1of2 problems solved to migrate from google

If the worker errors it wont delete the messages from the queue so they would get retried eventually

Similar Threads

Similar Threads

Similar Threads