I am seeing interesting issues with workflows. When I try to return simple object {} (empty object)

Bboop Durable Objects with alarms that call a worker every x seconds? Is your sequenti...

I

Isaac Hinman•4/30/25, 8:28 AM

Sequential requirement is per-user, not for the entire queue. Actually we'd ideally have multiple guaranteed sequential workflows per user

I

Isaac Hinman•4/30/25, 8:29 AM

For a bit of context, we're doing IMAP stuff and it's nasty – guaranteed sequential is unfortunately a necessity

IIsaac Hinman Sequential requirement is per-user, not for the entire queue. Actually we'd idea...

B

boop•4/30/25, 8:55 AM

In that case you could just use workflows and a queue if you expect to go over the instance creation limit.

The workflow could consume the jobs you've added in the queue, and the steps inside the workflow would be your sequential processing per job

O

Ollie•4/30/25, 1:07 PM

Is anyone experiencing any issues with workflows getting stuck, l have some workflows that have around 165 instances stuck in running state, showing 0 steps in the UI, and l can't pause or terminate them

D

DanGamble•4/30/25, 1:16 PM

I've been getting workflow issues today with deployment and in the UI as well

D

DanGamble•4/30/25, 1:16 PM

Uploaded oka-app (13.95 sec)                                                                                                                                                                                                                                                         │14:14:40 │                   typegen │   ℹ Watching for changes in /Users/dangamble/Code/oka-app...
                                                                                                                                                                                                                                                                                     │
✘ [ERROR] A request to the Cloudflare API (/accounts/###/workflows/pickup-points-post-create-workflow) failed.                                                                                                                                          │──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                                                                                                                                                                                                     │
  workflows.api.error.internal_server [code: 10001]

Uploaded oka-app (13.95 sec)                                                                                                                                                                                                                                                         │14:14:40 │                   typegen │   ℹ Watching for changes in /Users/dangamble/Code/oka-app...
                                                                                                                                                                                                                                                                                     │
✘ [ERROR] A request to the Cloudflare API (/accounts/###/workflows/pickup-points-post-create-workflow) failed.                                                                                                                                          │──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                                                                                                                                                                                                     │
  workflows.api.error.internal_server [code: 10001]

Uploaded oka-app (13.95 sec)                                                                                                                                                                                                                                                         │14:14:40 │                   typegen │   ℹ Watching for changes in /Users/dangamble/Code/oka-app...
                                                                                                                                                                                                                                                                                     │
✘ [ERROR] A request to the Cloudflare API (/accounts/###/workflows/pickup-points-post-create-workflow) failed.                                                                                                                                          │──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                                                                                                                                                                                                     │
  workflows.api.error.internal_server [code: 10001]

Uploaded oka-app (13.95 sec)                                                                                                                                                                                                                                                         │14:14:40 │                   typegen │   ℹ Watching for changes in /Users/dangamble/Code/oka-app...
                                                                                                                                                                                                                                                                                     │
✘ [ERROR] A request to the Cloudflare API (/accounts/###/workflows/pickup-points-post-create-workflow) failed.                                                                                                                                          │──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                                                                                                                                                                                                     │
  workflows.api.error.internal_server [code: 10001]

O

Ollie•4/30/25, 1:32 PM

Yeah everything has just ground to a halt for me, and the workflow api does nothing, so l can't stop or pause anything

B

Boragorn•4/30/25, 4:12 PM

Can someone help me with this?

O

Ollie•4/30/25, 4:13 PM

There is 100% an issue with workflows right now, everything just errors with

workflows.api.error.internal_server [code: 10001]

workflows.api.error.internal_server [code: 10001]

OOllie There is 100% an issue with workflows right now, everything just errors with `wo...

J

James•4/30/25, 4:26 PM

Could you /link so I can easily escalate this and the team can take a look at your account please?

JJames Could you /link so I can easily escalate this and the team can take a look at yo...

O

Ollie•4/30/25, 4:27 PM

Done!

J

James•4/30/25, 4:28 PM

Thanks!

OOllie There is 100% an issue with workflows right now, everything just errors with `wo...

D

Diogo Ferreira•4/30/25, 4:35 PM

can you please share your account id?

BBoragorn Can someone help me with this?

D

Diogo Ferreira•4/30/25, 4:35 PM

can you please share your account id?

DDiogo Ferreira can you please share your account id?

O

Ollie•4/30/25, 4:43 PM

I think my ID is: 0601cdee659d3394e0d37db62e73705c

DDiogo Ferreira can you please share your account id?

B

Boragorn•4/30/25, 5:08 PM

998ad664b89c2f9efa99a3b294eadd9d

OOllie I think my ID is: 0601cdee659d3394e0d37db62e73705c

D

Diogo Ferreira•4/30/25, 5:47 PM

Let us take a look into it

DDiogo Ferreira Let us take a look into it

O

Ollie•4/30/25, 8:49 PM

Just wondering if any updates on this?

O

Ollie•4/30/25, 9:05 PM

Just as an FYI the CF Dashboard is still having the same issues:

S

scook•4/30/25, 9:12 PM

This might not be related to the recent breakage, but after deploying my workflow to my staging/prod environments, I can see they exist in the Workflows dashboard, but calling them in the code results in:

Error: (workflow.not_found) Provided Workflow does not exist
  at callFetcher(cloudflare-internal:workflows-api:24:15)
  at WorkflowImpl.create(cloudflare-internal:workflows-api:78:24)

Error: (workflow.not_found) Provided Workflow does not exist
  at callFetcher(cloudflare-internal:workflows-api:24:15)
  at WorkflowImpl.create(cloudflare-internal:workflows-api:78:24)

Error: (workflow.not_found) Provided Workflow does not exist
  at callFetcher(cloudflare-internal:workflows-api:24:15)
  at WorkflowImpl.create(cloudflare-internal:workflows-api:78:24)

Error: (workflow.not_found) Provided Workflow does not exist
  at callFetcher(cloudflare-internal:workflows-api:24:15)
  at WorkflowImpl.create(cloudflare-internal:workflows-api:78:24)

Any idea what could be wrong?

Sscook This might not be related to the recent breakage, but after deploying my workflo...

S

scook•4/30/25, 9:21 PM

It appears to be a bug? This happened multiple times. When I first deployed a new Workflow to an environment, the code invocation would result in

Error: (workflow.not_found) Provided Workflow does not exist

Error: (workflow.not_found) Provided Workflow does not exist

. Upon the second redeployment, it would work correctly.

B

Boragorn•4/30/25, 10:23 PM

I might have identified a bug related to waitForEvent.

Here's the relevant code:

export class SimpleWorkflow extends WorkflowEntrypoint<
  Env,
  SimpleWorkflowPayload
> {
  async run(event: WorkflowEvent<SimpleWorkflowPayload>, step: WorkflowStep) {
    console.log("Simple workflow started", event.payload);

    await step.do("testing timeout", async () => {
      console.log(`Initializing test: ${event.payload.name}`);
      const eventResponse = await step.waitForEvent("waiting for event", {
        type: "test",
        timeout: 10000,
      });
      console.log("Test event received", eventResponse);
      return { status: "initialized" };
    });

    console.log("workflow complete");
  }
}

export class SimpleWorkflow extends WorkflowEntrypoint<
  Env,
  SimpleWorkflowPayload
> {
  async run(event: WorkflowEvent<SimpleWorkflowPayload>, step: WorkflowStep) {
    console.log("Simple workflow started", event.payload);

    await step.do("testing timeout", async () => {
      console.log(`Initializing test: ${event.payload.name}`);
      const eventResponse = await step.waitForEvent("waiting for event", {
        type: "test",
        timeout: 10000,
      });
      console.log("Test event received", eventResponse);
      return { status: "initialized" };
    });

    console.log("workflow complete");
  }
}

export class SimpleWorkflow extends WorkflowEntrypoint<
  Env,
  SimpleWorkflowPayload
> {
  async run(event: WorkflowEvent<SimpleWorkflowPayload>, step: WorkflowStep) {
    console.log("Simple workflow started", event.payload);

    await step.do("testing timeout", async () => {
      console.log(`Initializing test: ${event.payload.name}`);
      const eventResponse = await step.waitForEvent("waiting for event", {
        type: "test",
        timeout: 10000,
      });
      console.log("Test event received", eventResponse);
      return { status: "initialized" };
    });

    console.log("workflow complete");
  }
}

export class SimpleWorkflow extends WorkflowEntrypoint<
  Env,
  SimpleWorkflowPayload
> {
  async run(event: WorkflowEvent<SimpleWorkflowPayload>, step: WorkflowStep) {
    console.log("Simple workflow started", event.payload);

    await step.do("testing timeout", async () => {
      console.log(`Initializing test: ${event.payload.name}`);
      const eventResponse = await step.waitForEvent("waiting for event", {
        type: "test",
        timeout: 10000,
      });
      console.log("Test event received", eventResponse);
      return { status: "initialized" };
    });

    console.log("workflow complete");
  }
}

Whenever

waitForEvent

waitForEvent

fails due to a timeout exception, the step is retried as expected.

However, once the step re-runs, I’m no longer able to send events—the workflow appears to be running, but the event cannot be received or triggered anymore.

P

piffie•5/1/25, 5:41 AM

We hit a new error message:

Workflow V2: Error processing batch 5 {
  "message": "Aborting engine: Grace period complete",
  "error": {
    "remote": true,
    "durableObjectReset": true
  },

Workflow V2: Error processing batch 5 {
  "message": "Aborting engine: Grace period complete",
  "error": {
    "remote": true,
    "durableObjectReset": true
  },

Workflow V2: Error processing batch 5 {
  "message": "Aborting engine: Grace period complete",
  "error": {
    "remote": true,
    "durableObjectReset": true
  },

Workflow V2: Error processing batch 5 {
  "message": "Aborting engine: Grace period complete",
  "error": {
    "remote": true,
    "durableObjectReset": true
  },

is that due to worker updates during execution? its not very clear what this means.
Or is it we hit a limit? but what limit?

O

Ollie•5/1/25, 7:41 AM

I am not sure what has happened with workflows product recently, but since yesterday, the product just doesn't work for me, i've had to stop everything, and now looking at alternative solutions.

I can't trigger / pause / resume / terminate any workflows, l met with the error code

as earlier.

Over 3000 workflows failed yesterday with the exact same error

This instance experienced an internal error which prevented it from running. If this is persistent, please contact support: https://cfl.re/3WgEyrH

This instance experienced an internal error which prevented it from running. If this is persistent, please contact support: https://cfl.re/3WgEyrH

This instance experienced an internal error which prevented it from running. If this is persistent, please contact support: https://cfl.re/3WgEyrH

This instance experienced an internal error which prevented it from running. If this is persistent, please contact support: https://cfl.re/3WgEyrH

Any new workflow l try to start this morning goes straight into

queued

queued

state and will never run.

I have 4K+ messages going into dead-letter-queues because workflows can't be instantiated.

@James / @Diogo Ferreira any help would be appreciated, as this product was working perfectly up until yesterday moring.

P

piffie•5/1/25, 7:57 AM

And a question here:
is it better to split up heavy workflows into iterating through steps, or rather break it down that each pagination runs in a sub-workflow?

Ppiffie And a question here: is it better to split up heavy workflows into iterating th...

M

Matt Silverlock•5/1/25, 6:10 PM

Entirely depends on what you're doing? A single Workflow is (IMO) easier to reason about, keeps the code in the same place, and if the task is tightly bound, simplifies error boundaries and tracking

Start with one and (later) only add more if you need to fire off sub-Workflows.

SSakuraFC I am seeing interesting issues with workflows. When I try to return simple objec...

D

ddsgadget•5/1/25, 10:53 PM

Yes, we have the same issue and this seems to be a bug. All our workflows work perfectly fine, but throw exceptions in the dashboard after they finish. We generally return, {success:true} at the end of a workflow, which should be fine according the docs, but somehow this is an error? It's not clear what the actual format is for a return, especially the last step in a workflow. The only way around this for us has been to just terminate workflows at the end, otherwise they will all throw exceptions even if they work perfectly fine.

A

Austin B•5/1/25, 11:35 PM

I've got some intances of a workflow queued that don't seem to be getting picked up. Have been in this state for a while. Is there any way to tell in the dashboard why these aren't getting picked up?

Dddsgadget Yes, we have the same issue and this seems to be a bug. All our workflows work p...

S

SakuraFCOP•5/1/25, 11:49 PM

I found that the only reason that it throws (in the beginning for us) without impacting our workflows, is when we return an object or array from the step.

Ppiffie We hit a new error message: `Workflow V2: Error processing batch 5 { "message...

P

piffie•5/2/25, 3:14 AM

any info on that error @Matt Silverlock ?

AAustin B I've got some intances of a workflow queued that don't seem to be getting picked...

O

Ollie•5/2/25, 4:40 AM

You are most likely facing the same issues as am having in this thread: workflowsI am not sure what has happened with.

@Diogo Ferreira / @Thomas Ankcorn we seem to have a few more of the race condition issues

C

CrisOG•5/2/25, 4:04 PM

Hey there! I'm new to using Workflows and I've got a question.

I'm creating a workflow that depends on a mTLS certs binding for making requests. For some reason, mTLS certs bindings don't work locally (env.CERT is always empty) so I need to do

--remote

--remote

for it work.

The problem with that is that

--remote

--remote

also makes my workflow to run remotely as well. Ideally, I'd like to run workflows locally while still being able to have certs work.

Is there a way to make this work?

Thanks in advance!

CCrisOG Hey there! I'm new to using Workflows and I've got a question. I'm creating a w...

M

Matt Silverlock•5/2/25, 4:07 PM

I'll ask the team, but if the mTLS bindings aren't supported locally, then... there's not a way to make it work locally.

CCrisOG Hey there! I'm new to using Workflows and I've got a question. I'm creating a w...

M

Matt Silverlock•5/2/25, 4:10 PM

Update: yes, not supported locally, sorry. Working on better support across modes — some things like mTLS cert and/or Workers AI are hard to emulate locally.

MMatt Silverlock Update: yes, not supported locally, sorry. Working on better support across mode...

C

CrisOG•5/2/25, 4:14 PM

I don't mind if I need to run mTLS remotely. My goal is to use mTLS remotely while simultaneously running the workflow locally

Is that possible?

CCrisOG I don't mind if I need to run mTLS remotely. My goal is to use mTLS remotely whi...

M

Matt Silverlock•5/2/25, 4:40 PM

No, you can’t do this - that’s what I mean when you can’t make this work locally. We’re working on support for “mixed mode” to help solve but not yet ready.

MMatt Silverlock No, you can’t do this - that’s what I mean when you can’t make this work locally...

C

CrisOG•5/2/25, 4:41 PM

Thanks!

R

Rahul gangotri•5/3/25, 8:16 PM

How can we use cloudflare queue or workflow to process my users campaign where each campaign can send 50k to 100k messages

i want to schedule to a later date that i can do myself

i want to know if queue will be best or workflow will be best for this scenario

because i think if i send 100k messages then each msg in a queue will be a batch of 500 msg, then other users have to wait until 1 user campaign is over even if we have concurrency for 4-5

whereas in workflow we can invoke as many instances upto 4500 and queue other ones and each workflow can have multiple steps ( so i think it can run in parallel for 4500 campaigns)

what will be best? i am really confused, please help

RRahul gangotri How can we use cloudflare queue or workflow to process my users campaign where e...

T

tadhglewis•5/4/25, 6:22 AM

Pretty sure Workflows is built off of Queues. So read docs of both and weigh up pros and cons of your use case - e.g. auto retries etc

Ttadhglewis Pretty sure Workflows is built off of Queues. So read docs of both and weigh up ...

M

Matt Silverlock•5/4/25, 1:09 PM

It’s not, but it shouldn’t matter how it’s built. It’s a bit hard to understand the “why” behind what the other poster is asking though.

Sometimes you use both: Queues that can kick off Workflows to do the work on batches of <items>

RRahul gangotri How can we use cloudflare queue or workflow to process my users campaign where e...

O

Ollie•5/4/25, 5:01 PM

I would personally say workflows (and queues) are not ready for that kind of volume, they simply cant scale right now. Funnily enough workflows when they were in their BETA period were a pretty flawless platform, but since last week, only around 20% of our workflows ever actually run, around 50% get stuck, and then the other 30% don't even trigger (if you are using dead-letter-queues they end up getting filled up).

You will also never manage 4,500 instances running at the same time, we got up to say 350ish running at the same time, the rest just queued.

Until they fix the issues of scale, maybe consider another platform. Our company has basically had to turn some features off, that we built entirely on workflows and queues, and we are now considering something like temporal.

From my screenshot you can see we have over 5K stuck workflows, and we have around 3K+ messages in dead letter queues because workflows failed to even be instantiated. Before last week we ran nearly 1 million workflows in a month.

If they can get the issues sorted out, its probably the best workflow and queue platform l have used.

OOllie I would personally say workflows (and queues) are not ready for that kind of vol...

R

Rahul gangotri•5/4/25, 5:32 PM

, i am running bullmq for now buts its failing, otherwise i have to scale that or run kafka until this is resolved

I dont wanna use this until the issues are fixed otherwise i will end up having same issue after the migration

N

noskovvkirill•5/5/25, 12:23 AM

Hi guys! Is there any plan to add a middleware support to workflows? Similar to inngest (https://www.inngest.com/docs/features/middleware)? I need a persistent connection to my DB instance. Thanks

Sscook 🐛 It appears to be a bug? This happened multiple times. When I first deployed a...

T

tyx•5/5/25, 1:09 AM

I am getting a very confusing bug. I am getting.

Error: (workflow.not_found) Provided Workflow does not exist

Error: (workflow.not_found) Provided Workflow does not exist

when deployed but If I run it locally (but on remote mode) i detects the workflow and triggers it successfully. (i've tried re-deploying without success)

Is this a bug on CF side, seems like it?

T

tyx•5/5/25, 1:12 AM

shared my id with @Diogo Ferreira

I

Isak•5/5/25, 7:00 AM

When a workflow is awakened from sleep, does that count as a separate invocation for billing purposes?

I am seeing interesting issues with workflows. When I try to return simple object {} (empty object)

Similar Threads

Similar Threads

Similar Threads