Cloudflare Developers•2y ago

I'm not sure if my accountId is 100% correct, I got it from the url

AArthur🍃I'm not sure if my accountId is 100% correct, I got it from the url

Pranshu Maheshwari•6/25/24, 8:53 PM

Stepping in for Matt here, thanks! We're looking into this.

Pranshu Maheshwari•6/25/24, 9:30 PM

Hi @everyone! Pranshu here. I’m a product manager at Cloudflare and I’ve recently taken on Queues. I’d love to learn more about how all of you are using Queues, and how you'd like to see it improve.

I’d especially love to hear from folks building startups. I’ve been a 2x founder myself, so I’ve been in your shoes!

If you’d like to grab some time to chat in the next two weeks, here’s my calendar: https://calendar.app.google/pdzfMFmwACg2PN8w8

Appointments

zegevlier•6/25/24, 9:34 PM

Sorry everyone for the ping!

Erisa•6/25/24, 10:01 PM

Let's please keep messages on-topic - mistakes happen, everyone is sorry about it - lets move on. If you want to comment about things not related to queues, try #off-topic

PPranshu Maheshwari Hi @everyone! Pranshu here. I’m a product manager at Cloudflare and I’ve recentl...

quisi.do•6/25/24, 10:11 PM

I'm working on a startup, and I'm unable to use queues for proof-of-concepts without first upgrading to a paid plan. I'm currently working free tier for PoCs and don't want to invest financially into a solution service until revenue is generated. Any chance of getting a free tier for queues?

Avat•6/25/24, 10:14 PM

Hello, I'm looking for advice how to handle canceled request.
So, I'm using an API which sends array of objects to my worker, which then iterates through this array and sends every object to queue.

The problem is, this API closes the connection almost immediately and the worker gets canceled.

How can I keep worker up until whole object is being sent as messages to queue?

Salamus•6/25/24, 10:20 PM

PPranshu Maheshwari Hi @everyone! Pranshu here. I’m a product manager at Cloudflare and I’ve recentl...

TheMightyPenguin•6/25/24, 10:34 PM

Queues not working on remote dev is a friction point for me! I mostly do all of my local dev without using wrangler dev remote, but as I’m working on an app that uses browser rendering, I have had to switch to remote as browser rendering does not work locally.

I could also phrase this problem as browser rendering not working locally tho, is that coming at some point? Everything else feels frictionless locally

Qquisi.do I'm working on a startup, and I'm unable to use queues for proof-of-concepts wit...

Pranshu Maheshwari•6/25/24, 10:34 PM

there is a free tier (up to 1mm operations / month), but yes, you do need to be on the workers paid plan to use it

Qquisi.do I'm working on a startup, and I'm unable to use queues for proof-of-concepts wit...

grandell•6/25/24, 10:35 PM

If you can afford $5 than you should upgrade. I work a lot of services and if you use it properly you can very easily make your money's worth.

Ggrandell If you can afford $5 than you should upgrade. I work a lot of services and if yo...

quisi.do•6/25/24, 11:07 PM

I have no doubt that I will benefit from it after having services; but right now, it is impossible to build PoCs for services with queues. My goal is to build a service on free tier, establish market, then upgrade to increase my limits to account for the new traffic. But without queues in my free tier service, I am unable to launch a new service that harnesses queues.

Qquisi.do I have no doubt that I will benefit from it after having services; but right now...

caleb•6/25/24, 11:51 PM

Welcome to running a business. Sometimes you need to spend before you have revenue.

Paul•6/26/24, 12:46 AM

When writing multiple queue consumers, they always need to be defined in the root default export right? So if we have multiple queues in one worker, the only way to define multiple consumers is to have a switch statement or equivalent in the root queue-key export and route the message to the right consumer right? One would assume I would need to have a "name" or equivalent key in the message body as a result, in order to route the message to the correct consumer right?

dot.•6/26/24, 1:22 AM

how does queues compare to Amazon sqs?

PPranshu Maheshwari Hi @everyone! Pranshu here. I’m a product manager at Cloudflare and I’ve recentl...

LePichuDoesDev•6/26/24, 2:20 AM

LLog i dont use queues because my site is bad

Ducky•6/26/24, 4:18 AM

PPranshu Maheshwari Hi @everyone! Pranshu here. I’m a product manager at Cloudflare and I’ve recentl...

Ducky•6/26/24, 4:18 AM

thing is, i dont :dogekek:

PPranshu Maheshwari Hi @everyone! Pranshu here. I’m a product manager at Cloudflare and I’ve recentl...

Paul•6/26/24, 4:19 AM

I’m using queues to batch download media files for my saas. I just wonder the rationale for why there’s only one queue consumer function and it needs to be defined in the root.

It’s fine because we can still route the message to the correct queue consumers by some name or type key inside the message body to but I’m curious about the rationale for this architecture… unless this is just common and I’m just not familiar with it

PPranshu Maheshwari Hi @everyone! Pranshu here. I’m a product manager at Cloudflare and I’ve recentl...

SteffTek•6/26/24, 4:38 AM

We are building a fully digital, mobile point of sales system and are using queues for asynchronous jobs, like receipt creation, regulatory required backup jobs, syncing to other APIs and cash point closings.

Without queues we wouldn't be able to do this since some requests take longer than 30s.

Edit: we have a crazy technology stack. CF as backend, flutter for the mobile app (so far so good) and hecking Delphi for the windows env :pain: best startup ever

PPranshu Maheshwari Hi @everyone! Pranshu here. I’m a product manager at Cloudflare and I’ve recentl...

DeChill•6/26/24, 6:53 AM

I'm a big fan of CF Queues and especially the Consumer Worker model! The biggest thing I wish for is easier way to publish items to a Queue from external systems (like external Kafka Topics).

For example I'm using Debezium for change data capture on an external Database.
This then leads me to having to use Kafka -> HTTP -> Worker -> Queue to get these messages to a CF Queue which is a lot of overhead and you have to deal with all of the usual Webhook headaches

Original message was deleted

DaniFoldi•6/26/24, 7:19 AM

Thanks

Original message was deleted

Ɽ

ⱤØ₥₳₦•6/26/24, 7:33 AM

What is it and how we can use it? what's the real life usecase of this?

DDeChill I'm a big fan of CF Queues and especially the Consumer Worker model! The biggest...

oof2win2•6/26/24, 8:00 AM

i heavily agree with this. queues have a intended scenario of working on a per-customer basis, which i understand to be one queue per customer - each customer can then be a HTTP consumer and get+ack messages, but the issue is publishing messages to the queue. creating a queue is already possible with the CF api, the question only is if you can create enough access tokens for the queues

carlos-alberto•6/26/24, 9:49 AM

Since ~18:00 yesterday one of our queues has been erroring with Queue send failed: Internal Server ErrorQueue send failed: Internal Server Error.

Trying to redeploy the worker gives workers.api.error.unknown [code: 10013]workers.api.error.unknown [code: 10013], as does trying to delete the queue. Anyone able to help figure out what is going on here?

Queue ID: 18dbd75d6a1946b48f614689ed23bec9

Ccarlos-alberto Since ~18:00 yesterday one of our queues has been erroring with `Queue send fail...

Matt Silverlock•6/26/24, 10:27 AM

Which timezone and what is your account ID? (safe to post here)

MMatt Silverlock Which timezone and what is your account ID? (safe to post here)

carlos-alberto•6/26/24, 11:02 AM

Timezone: BST
Account ID: ef98901aa25fef025cee6651b601b491

Ccarlos-alberto Timezone: BST Account ID: ef98901aa25fef025cee6651b601b491

Matt Silverlock•6/26/24, 12:27 PM

Are you still seeing issues? (either sending messages or creating a Queue)

Was this sporadic?

MMatt Silverlock Are you still seeing issues? (either sending messages or creating a Queue) Was...

carlos-alberto•6/26/24, 12:43 PM

This is ongoing, we are unable to interact with that queue in any way. Even trying to update the "Delivery delay" gives the same error

Ccarlos-alberto This is ongoing, we are unable to interact with that queue in any way. Even tryi...

Matt Silverlock•6/26/24, 12:44 PM

@Pranshu Maheshwari ^

carlos-alberto•6/26/24, 12:44 PM

We've deployed a new version of our worker under a new name, and a new queue so we can work around this, but it's ongoing for this queue

Pranshu Maheshwari•6/26/24, 1:27 PM

Thanks for highlighting this, we're looking into it

Original message was deleted

santosh•6/26/24, 4:58 PM

@FraznoFire ?

PPranshu Maheshwari Hi @everyone! Pranshu here. I’m a product manager at Cloudflare and I’ve recentl...

Fabian•6/26/24, 5:25 PM

We use queues to aggregate calls to an external API, only thing that would make queues for that use-case better is that batches currently cannot be grouped by a (custom) identifier - say we scope all the data to a single customer and group it under a 30s queue timeframe, we could have more efficient calls to the external API (since they also operate under a per-customer/ID resource scope).
This is relevant because the external API has relatively strict account rate-limits (not scoped to the ID).
Right now it's hit or miss since possibly 100 different customers could join the same batch effectively resulting in groups of 1 and thus 100 API calls.
In this case we would prefer grouped batching over quick execution of a random batch if you understand what I mean

andyy•6/26/24, 5:49 PM

is there any kind of queue bindings-per-worker limit? (sorry if I missed it anywhere in the docs). curious whether it's possible/sensible to have a queue-per-customer architecture all written to by the same producer worker. thanks!

Ttizmagik Hey folks. On latest wrangler (`3.62.0`) and it looks like trying to do `wrangle...

Pranshu Maheshwari•6/26/24, 8:32 PM

Hmm I haven't been able to replicate this on my side. CLI arguments should override what you've specified in wrangler.tomlwrangler.toml

Thisjrodriguez•6/27/24, 6:28 AM

Hello,

I intend to use queues to save the data of each visit and then import them to have an analytic.

Is this the use of queues? I understand that yes, but if someone confirms me I would appreciate it.

TThisjrodriguez Hello, I intend to use queues to save the data of each visit and then import th...

Matt Silverlock•6/27/24, 11:19 AM

What kind of analytics? Writing directly to WAE may be more effective: https://developers.cloudflare.com/analytics/analytics-engine/

Cloudflare Docs

Workers Analytics Engine · Cloudflare Analytics docs

Workers Analytics Engine provides unlimited-cardinality analytics at scale, via a built-in API to write data points from Workers, and a SQL API to …

MMatt Silverlock What kind of analytics? Writing directly to WAE may be more effective: https://d...

Thisjrodriguez•6/27/24, 11:47 AM

a personalized one, I just wanted to know if it is useful for what I proposed.

TThisjrodriguez a personalized one, I just wanted to know if it is useful for what I proposed.

Matt Silverlock•6/27/24, 11:50 AM

It’s hard to say: you haven’t provided enough detail.

A Queue simply puts messages onto a queue for you to consume later.

That can be useful for deferring work or batching tasks together for efficiency, and/or getting “slow” API calls off the hot path so they don’t slow down user-facing responses.

I'm not sure if my accountId is 100% correct, I got it from the url

Similar Threads