Durable Objects with alarms that call a worker every x seconds? Is your sequential requirement per-u
Durable Objects with alarms that call a worker every x seconds? Is your sequential requirement per-user or for the entire work/jobs queue?
SI am seeing interesting issues with workflows. When I try to return simple object {} (empty object) as a state, workflow always logs exception but does work. Took me half a day to try and figure out what was going wrong despite it working. So far the only workaround I found is to return stringified object and from the step and then convert it back to the object and use in the remainder of the workflow. Not sure if it's a bug or expected.
ISequential requirement is per-user, not for the entire queue. Actually we'd ideally have multiple guaranteed sequential workflows per user
IFor a bit of context, we're doing IMAP stuff and it's nasty – guaranteed sequential is unfortunately a necessity
BIn that case you could just use workflows and a queue if you expect to go over the instance creation limit.
The workflow could consume the jobs you've added in the queue, and the steps inside the workflow would be your sequential processing per job
The workflow could consume the jobs you've added in the queue, and the steps inside the workflow would be your sequential processing per job
OIs anyone experiencing any issues with workflows getting stuck, l have some workflows that have around 165 instances stuck in running state, showing 0 steps in the UI, and l can't pause or terminate them
DI've been getting workflow issues today with deployment and in the UI as well
DOYeah everything has just ground to a halt for me, and the workflow api does nothing, so l can't stop or pause anything
BCan someone help me with this?
OThere is 100% an issue with workflows right now, everything just errors with
workflows.api.error.internal_server [code: 10001]
JCould you /link so I can easily escalate this and the team can take a look at your account please?
ODone!
JThanks!
Dcan you please share your account id?
Dcan you please share your account id?
OI think my ID is: 0601cdee659d3394e0d37db62e73705c
B998ad664b89c2f9efa99a3b294eadd9d
DLet us take a look into it
OJust wondering if any updates on this?
OJust as an FYI the CF Dashboard is still having the same issues:
SThis might not be related to the recent breakage, but after deploying my workflow to my staging/prod environments, I can see they exist in the Workflows dashboard, but calling them in the code results in:
Any idea what could be wrong?
Any idea what could be wrong?
S
It appears to be a bug? This happened multiple times. When I first deployed a new Workflow to an environment, the code invocation would result in Error: (workflow.not_found) Provided Workflow does not existI might have identified a bug related to waitForEvent.
Here's the relevant code:
Whenever
However, once the step re-runs, I’m no longer able to send events—the workflow appears to be running, but the event cannot be received or triggered anymore.
Here's the relevant code:
Whenever
waitForEventHowever, once the step re-runs, I’m no longer able to send events—the workflow appears to be running, but the event cannot be received or triggered anymore.
PWe hit a new error message:
is that due to worker updates during execution? its not very clear what this means.
Or is it we hit a limit? but what limit?
Workflow V2: Error processing batch 5 {
"message": "Aborting engine: Grace period complete",
"error": {
"remote": true,
"durableObjectReset": true
},is that due to worker updates during execution? its not very clear what this means.
Or is it we hit a limit? but what limit?
OI am not sure what has happened with workflows product recently, but since yesterday, the product just doesn't work for me, i've had to stop everything, and now looking at alternative solutions.
I can't trigger / pause / resume / terminate any workflows, l met with the error code
Over 3000 workflows failed yesterday with the exact same error
Any new workflow l try to start this morning goes straight into
I have 4K+ messages going into dead-letter-queues because workflows can't be instantiated.
@James / @Diogo Ferreira any help would be appreciated, as this product was working perfectly up until yesterday moring.
I can't trigger / pause / resume / terminate any workflows, l met with the error code
10001Over 3000 workflows failed yesterday with the exact same error
This instance experienced an internal error which prevented it from running. If this is persistent, please contact support: https://cfl.re/3WgEyrHAny new workflow l try to start this morning goes straight into
queuedI have 4K+ messages going into dead-letter-queues because workflows can't be instantiated.
@James / @Diogo Ferreira any help would be appreciated, as this product was working perfectly up until yesterday moring.
PAnd a question here:
is it better to split up heavy workflows into iterating through steps, or rather break it down that each pagination runs in a sub-workflow?
is it better to split up heavy workflows into iterating through steps, or rather break it down that each pagination runs in a sub-workflow?
MEntirely depends on what you're doing? A single Workflow is (IMO) easier to reason about, keeps the code in the same place, and if the task is tightly bound, simplifies error boundaries and tracking
Start with one and (later) only add more if you need to fire off sub-Workflows.
Start with one and (later) only add more if you need to fire off sub-Workflows.
DYes, we have the same issue and this seems to be a bug. All our workflows work perfectly fine, but throw exceptions in the dashboard after they finish. We generally return, {success:true} at the end of a workflow, which should be fine according the docs, but somehow this is an error? It's not clear what the actual format is for a return, especially the last step in a workflow. The only way around this for us has been to just terminate workflows at the end, otherwise they will all throw exceptions even if they work perfectly fine.
AI've got some intances of a workflow queued that don't seem to be getting picked up. Have been in this state for a while. Is there any way to tell in the dashboard why these aren't getting picked up?
SI found that the only reason that it throws (in the beginning for us) without impacting our workflows, is when we return an object or array from the step.
Pany info on that error @Matt Silverlock ?
OYou are most likely facing the same issues as am having in this thread: workflowsI am not sure what has happened with.
@Diogo Ferreira / @Thomas Ankcorn we seem to have a few more of the race condition issues
@Diogo Ferreira / @Thomas Ankcorn we seem to have a few more of the race condition issues
CHey there! I'm new to using Workflows and I've got a question.
I'm creating a workflow that depends on a mTLS certs binding for making requests. For some reason, mTLS certs bindings don't work locally (env.CERT is always empty) so I need to do
The problem with that is that
Is there a way to make this work?
Thanks in advance!
I'm creating a workflow that depends on a mTLS certs binding for making requests. For some reason, mTLS certs bindings don't work locally (env.CERT is always empty) so I need to do
--remoteThe problem with that is that
--remoteIs there a way to make this work?
Thanks in advance!
MI'll ask the team, but if the mTLS bindings aren't supported locally, then... there's not a way to make it work locally.
MUpdate: yes, not supported locally, sorry. Working on better support across modes — some things like mTLS cert and/or Workers AI are hard to emulate locally.
CI don't mind if I need to run mTLS remotely. My goal is to use mTLS remotely while simultaneously running the workflow locally
Is that possible?
Is that possible?
MNo, you can’t do this - that’s what I mean when you can’t make this work locally. We’re working on support for “mixed mode” to help solve but not yet ready.
RHow can we use cloudflare queue or workflow to process my users campaign where each campaign can send 50k to 100k messages
i want to schedule to a later date that i can do myself
i want to know if queue will be best or workflow will be best for this scenario
because i think if i send 100k messages then each msg in a queue will be a batch of 500 msg, then other users have to wait until 1 user campaign is over even if we have concurrency for 4-5
whereas in workflow we can invoke as many instances upto 4500 and queue other ones and each workflow can have multiple steps ( so i think it can run in parallel for 4500 campaigns)
what will be best? i am really confused, please help
i want to schedule to a later date that i can do myself
i want to know if queue will be best or workflow will be best for this scenario
because i think if i send 100k messages then each msg in a queue will be a batch of 500 msg, then other users have to wait until 1 user campaign is over even if we have concurrency for 4-5
whereas in workflow we can invoke as many instances upto 4500 and queue other ones and each workflow can have multiple steps ( so i think it can run in parallel for 4500 campaigns)
what will be best? i am really confused, please help
TPretty sure Workflows is built off of Queues. So read docs of both and weigh up pros and cons of your use case - e.g. auto retries etc
MIt’s not, but it shouldn’t matter how it’s built. It’s a bit hard to understand the “why” behind what the other poster is asking though.
Sometimes you use both: Queues that can kick off Workflows to do the work on batches of <items>
Sometimes you use both: Queues that can kick off Workflows to do the work on batches of <items>
OI would personally say workflows (and queues) are not ready for that kind of volume, they simply cant scale right now. Funnily enough workflows when they were in their BETA period were a pretty flawless platform, but since last week, only around 20% of our workflows ever actually run, around 50% get stuck, and then the other 30% don't even trigger (if you are using dead-letter-queues they end up getting filled up).
You will also never manage 4,500 instances running at the same time, we got up to say 350ish running at the same time, the rest just queued.
Until they fix the issues of scale, maybe consider another platform. Our company has basically had to turn some features off, that we built entirely on workflows and queues, and we are now considering something like temporal.
From my screenshot you can see we have over 5K stuck workflows, and we have around 3K+ messages in dead letter queues because workflows failed to even be instantiated. Before last week we ran nearly 1 million workflows in a month.
If they can get the issues sorted out, its probably the best workflow and queue platform l have used.
You will also never manage 4,500 instances running at the same time, we got up to say 350ish running at the same time, the rest just queued.
Until they fix the issues of scale, maybe consider another platform. Our company has basically had to turn some features off, that we built entirely on workflows and queues, and we are now considering something like temporal.
From my screenshot you can see we have over 5K stuck workflows, and we have around 3K+ messages in dead letter queues because workflows failed to even be instantiated. Before last week we ran nearly 1 million workflows in a month.
If they can get the issues sorted out, its probably the best workflow and queue platform l have used.
R
, i am running bullmq for now buts its failing, otherwise i have to scale that or run kafka until this is resolvedI dont wanna use this until the issues are fixed otherwise i will end up having same issue after the migration
NHi guys! Is there any plan to add a middleware support to workflows? Similar to inngest (https://www.inngest.com/docs/features/middleware)? I need a persistent connection to my DB instance. Thanks
TI am getting a very confusing bug. I am getting.
Is this a bug on CF side, seems like it?
Error: (workflow.not_found) Provided Workflow does not existIs this a bug on CF side, seems like it?
Tshared my id with @Diogo Ferreira
