We hit a new error message: `Workflow V2: Error processing batch 5 { "message": "Aborting engine:

We hit a new error message:

Workflow V2: Error processing batch 5 {
  "message": "Aborting engine: Grace period complete",
  "error": {
    "remote": true,
    "durableObjectReset": true
  },

Workflow V2: Error processing batch 5 {
  "message": "Aborting engine: Grace period complete",
  "error": {
    "remote": true,
    "durableObjectReset": true
  },

Workflow V2: Error processing batch 5 {
  "message": "Aborting engine: Grace period complete",
  "error": {
    "remote": true,
    "durableObjectReset": true
  },

Workflow V2: Error processing batch 5 {
  "message": "Aborting engine: Grace period complete",
  "error": {
    "remote": true,
    "durableObjectReset": true
  },

is that due to worker updates during execution? its not very clear what this means.
Or is it we hit a limit? but what limit?

Ollie•5/1/25, 7:41 AM

I am not sure what has happened with workflows product recently, but since yesterday, the product just doesn't work for me, i've had to stop everything, and now looking at alternative solutions.

I can't trigger / pause / resume / terminate any workflows, l met with the error code

as earlier.

Over 3000 workflows failed yesterday with the exact same error

This instance experienced an internal error which prevented it from running. If this is persistent, please contact support: https://cfl.re/3WgEyrH

This instance experienced an internal error which prevented it from running. If this is persistent, please contact support: https://cfl.re/3WgEyrH

Any new workflow l try to start this morning goes straight into

queued

queued

state and will never run.

I have 4K+ messages going into dead-letter-queues because workflows can't be instantiated.

@James / @Diogo Ferreira any help would be appreciated, as this product was working perfectly up until yesterday moring.

piffieOP•5/1/25, 7:57 AM

And a question here:
is it better to split up heavy workflows into iterating through steps, or rather break it down that each pagination runs in a sub-workflow?

Ppiffie And a question here: is it better to split up heavy workflows into iterating th...

Matt Silverlock•5/1/25, 6:10 PM

Entirely depends on what you're doing? A single Workflow is (IMO) easier to reason about, keeps the code in the same place, and if the task is tightly bound, simplifies error boundaries and tracking

Start with one and (later) only add more if you need to fire off sub-Workflows.

SSakuraFC I am seeing interesting issues with workflows. When I try to return simple objec...

ddsgadget•5/1/25, 10:53 PM

Yes, we have the same issue and this seems to be a bug. All our workflows work perfectly fine, but throw exceptions in the dashboard after they finish. We generally return, {success:true} at the end of a workflow, which should be fine according the docs, but somehow this is an error? It's not clear what the actual format is for a return, especially the last step in a workflow. The only way around this for us has been to just terminate workflows at the end, otherwise they will all throw exceptions even if they work perfectly fine.

Austin B•5/1/25, 11:35 PM

I've got some intances of a workflow queued that don't seem to be getting picked up. Have been in this state for a while. Is there any way to tell in the dashboard why these aren't getting picked up?

Dddsgadget Yes, we have the same issue and this seems to be a bug. All our workflows work p...

SakuraFC•5/1/25, 11:49 PM

I found that the only reason that it throws (in the beginning for us) without impacting our workflows, is when we return an object or array from the step.

Ppiffie We hit a new error message: `Workflow V2: Error processing batch 5 { "message...

piffieOP•5/2/25, 3:14 AM

any info on that error @Matt Silverlock ?

AAustin B I've got some intances of a workflow queued that don't seem to be getting picked...

Ollie•5/2/25, 4:40 AM

You are most likely facing the same issues as am having in this thread: workflowsI am not sure what has happened with.

@Diogo Ferreira / @Thomas Ankcorn we seem to have a few more of the race condition issues

CrisOG•5/2/25, 4:04 PM

Hey there! I'm new to using Workflows and I've got a question.

I'm creating a workflow that depends on a mTLS certs binding for making requests. For some reason, mTLS certs bindings don't work locally (env.CERT is always empty) so I need to do

--remote

--remote

for it work.

The problem with that is that

--remote

--remote

also makes my workflow to run remotely as well. Ideally, I'd like to run workflows locally while still being able to have certs work.

Is there a way to make this work?

Thanks in advance!

CCrisOG Hey there! I'm new to using Workflows and I've got a question. I'm creating a w...

Matt Silverlock•5/2/25, 4:07 PM

I'll ask the team, but if the mTLS bindings aren't supported locally, then... there's not a way to make it work locally.

CCrisOG Hey there! I'm new to using Workflows and I've got a question. I'm creating a w...

Matt Silverlock•5/2/25, 4:10 PM

Update: yes, not supported locally, sorry. Working on better support across modes — some things like mTLS cert and/or Workers AI are hard to emulate locally.

MMatt Silverlock Update: yes, not supported locally, sorry. Working on better support across mode...

CrisOG•5/2/25, 4:14 PM

I don't mind if I need to run mTLS remotely. My goal is to use mTLS remotely while simultaneously running the workflow locally

Is that possible?

CCrisOG I don't mind if I need to run mTLS remotely. My goal is to use mTLS remotely whi...

Matt Silverlock•5/2/25, 4:40 PM

No, you can’t do this - that’s what I mean when you can’t make this work locally. We’re working on support for “mixed mode” to help solve but not yet ready.

MMatt Silverlock No, you can’t do this - that’s what I mean when you can’t make this work locally...

CrisOG•5/2/25, 4:41 PM

Thanks!

Rahul gangotri•5/3/25, 8:16 PM

How can we use cloudflare queue or workflow to process my users campaign where each campaign can send 50k to 100k messages

i want to schedule to a later date that i can do myself

i want to know if queue will be best or workflow will be best for this scenario

because i think if i send 100k messages then each msg in a queue will be a batch of 500 msg, then other users have to wait until 1 user campaign is over even if we have concurrency for 4-5

whereas in workflow we can invoke as many instances upto 4500 and queue other ones and each workflow can have multiple steps ( so i think it can run in parallel for 4500 campaigns)

what will be best? i am really confused, please help

RRahul gangotri How can we use cloudflare queue or workflow to process my users campaign where e...

tadhglewis•5/4/25, 6:22 AM

Pretty sure Workflows is built off of Queues. So read docs of both and weigh up pros and cons of your use case - e.g. auto retries etc

Ttadhglewis Pretty sure Workflows is built off of Queues. So read docs of both and weigh up ...

Matt Silverlock•5/4/25, 1:09 PM

It’s not, but it shouldn’t matter how it’s built. It’s a bit hard to understand the “why” behind what the other poster is asking though.

Sometimes you use both: Queues that can kick off Workflows to do the work on batches of <items>

RRahul gangotri How can we use cloudflare queue or workflow to process my users campaign where e...

Ollie•5/4/25, 5:01 PM

I would personally say workflows (and queues) are not ready for that kind of volume, they simply cant scale right now. Funnily enough workflows when they were in their BETA period were a pretty flawless platform, but since last week, only around 20% of our workflows ever actually run, around 50% get stuck, and then the other 30% don't even trigger (if you are using dead-letter-queues they end up getting filled up).

You will also never manage 4,500 instances running at the same time, we got up to say 350ish running at the same time, the rest just queued.

Until they fix the issues of scale, maybe consider another platform. Our company has basically had to turn some features off, that we built entirely on workflows and queues, and we are now considering something like temporal.

From my screenshot you can see we have over 5K stuck workflows, and we have around 3K+ messages in dead letter queues because workflows failed to even be instantiated. Before last week we ran nearly 1 million workflows in a month.

If they can get the issues sorted out, its probably the best workflow and queue platform l have used.

OOllie I would personally say workflows (and queues) are not ready for that kind of vol...

Rahul gangotri•5/4/25, 5:32 PM

, i am running bullmq for now buts its failing, otherwise i have to scale that or run kafka until this is resolved

I dont wanna use this until the issues are fixed otherwise i will end up having same issue after the migration

noskovvkirill•5/5/25, 12:23 AM

Hi guys! Is there any plan to add a middleware support to workflows? Similar to inngest (https://www.inngest.com/docs/features/middleware)? I need a persistent connection to my DB instance. Thanks

Sscook 🐛 It appears to be a bug? This happened multiple times. When I first deployed a...

tyx•5/5/25, 1:09 AM

I am getting a very confusing bug. I am getting.

Error: (workflow.not_found) Provided Workflow does not exist

Error: (workflow.not_found) Provided Workflow does not exist

when deployed but If I run it locally (but on remote mode) i detects the workflow and triggers it successfully. (i've tried re-deploying without success)

Is this a bug on CF side, seems like it?

tyx•5/5/25, 1:12 AM

shared my id with @Diogo Ferreira

Isak•5/5/25, 7:00 AM

When a workflow is awakened from sleep, does that count as a separate invocation for billing purposes?

Nnoskovvkirill Hi guys! Is there any plan to add a middleware support to workflows? Similar to ...

Matt Silverlock•5/5/25, 10:28 AM

A DB connection is alive for as long as the Workflow instance is running (short of errors or long sleep periods). I’m not sure you need middleware as we don’t function like Inngest does.

Hharisvsulaiman Does anyone have a problem when the workflow sometimes does not get the environm...

Matt Silverlock•5/5/25, 10:28 AM

There’s not enough info to go off here. Show your code and a reproduction.

Ttyx I am getting a very confusing bug. I am getting. `Error: (workflow.not_found) Pr...

tyx•5/5/25, 12:37 PM

@Matt does this seem like a system issue, or I am using things incorrectly. surprising that when run locally it works as expected and the workflow gets triggered without any errors. It's only shows up in the deployed version

Ttyx @Matt does this seem like a system issue, or I am using things incorrectly. surp...

Matt Silverlock•5/5/25, 12:56 PM

Does the Worker you deploy bind to it? Show your code.

Hharisvsulaiman Sure. Can I point you towards the cloudflare organisation?. This happens often. ...

Matt Silverlock•5/5/25, 2:03 PM

I can’t see your code unless you share a repro

MMatt Silverlock Does the Worker you deploy bind to it? Show your code.

tyx•5/5/25, 2:25 PM

UI shows so:
Following is my wranger.toml parts

name = "docpad-session-init"
main = "src/index.ts"
compatibility_date = "2024-10-22"
compatibility_flags = ["nodejs_compat"]

[[workflows]]
binding = "DOCPAD_SESSION_INIT"
name = "docpad-session-init"
class_name = "DocpadSessionInit"

name = "docpad-session-init"
main = "src/index.ts"
compatibility_date = "2024-10-22"
compatibility_flags = ["nodejs_compat"]

[[workflows]]
binding = "DOCPAD_SESSION_INIT"
name = "docpad-session-init"
class_name = "DocpadSessionInit"

same request made locally i get 200:
but making the same request to deployed version either through cf web ui or any other means i get hit with workflow not found.

7777777•5/5/25, 6:34 PM

hi all - wondering if anyone has any resources to help me get vitest working since adding

[[workflows]]

[[workflows]]

to my toml file?
I basically go the following config setup where I needed to disabled

isolatedStorage

isolatedStorage

otherwise it was complaining about not being able to find my imported Workflow class to bind.

import { defineWorkersConfig } from '@cloudflare/vitest-pool-workers/config';

export default defineWorkersConfig({
    test: {
        poolOptions: {
            workers: {
                isolatedStorage: false,
                wrangler: { configPath: './wrangler.toml' },
            },
            poolMatchGlobs: [],
        },
        maxConcurrency: 1,
    },
})

import { defineWorkersConfig } from '@cloudflare/vitest-pool-workers/config';

export default defineWorkersConfig({
    test: {
        poolOptions: {
            workers: {
                isolatedStorage: false,
                wrangler: { configPath: './wrangler.toml' },
            },
            poolMatchGlobs: [],
        },
        maxConcurrency: 1,
    },
})

but I'm unable to run even a simple test anymore due to

import { createExecutionContext } from "cloudflare:test";
import { describe, it, expect } from "vitest";
describe("test", () => {
  it('test', () => {
    // referencing something from cloudflare:test
    const ctx = createExecutionContext()
    expect(2).toBe(2)
  })
})

import { createExecutionContext } from "cloudflare:test";
import { describe, it, expect } from "vitest";
describe("test", () => {
  it('test', () => {
    // referencing something from cloudflare:test
    const ctx = createExecutionContext()
    expect(2).toBe(2)
  })
})

workerd/jsg/util.c++:320: error: e = kj/table.c++:49: failed: inserted row already exists in table

7777777•5/5/25, 6:45 PM

ah adding

singleWorkfer: true

singleWorkfer: true

got it working, so it must be that the workflows each spin up their own worker, and try to initialize something for each test?

SSakuraFC I found that the only reason that it throws (in the beginning for us) without im...

ddsgadget•5/5/25, 7:11 PM

right, but why can't you return an object or array? We routinely return an object or array and use it next steps. Always works. What are you supposed to return? How do you use data between steps if you can't return an array or object?

Dddsgadget right, but why can't you return an object or array? We routinely return an objec...

SakuraFC•5/5/25, 10:45 PM

It works for sure but it also throws an exception for some reason. The only way I found to stop the exception is to return string JSON.stringify and then use JSON.parse to turn it back to JS object/array

MMatt Silverlock A DB connection is alive for as long as the Workflow instance is running (short ...

noskovvkirill•5/6/25, 12:28 AM

Thanks @Matt! Is it safe do define connection outside of a step? Wouldn't it break the "Do not rely on state outside of a step" rule of Workflows?
We're connecting to our Postgres using connecting string.

const db = getDB(this.env);

const db = getDB(this.env);

Nnoskovvkirill Thanks @Matt! Is it safe do define connection outside of a step? Wouldn't it bre...

Matt Silverlock•5/6/25, 12:36 AM

I wouldn't consider that state - it's not data that you are trying to persist.

I would however do something like this, so that you can make sure the DB connection is alive / memoize the DB where you can:

// inside the run() method
const db: TypeOfYourDB | undefined

// inside the run() method
const db: TypeOfYourDB | undefined

// elsewhere, as a helper - e.g. on the Workflow class
async function getDB(db, this.env) {
   if (!db) {
     db = // create a new connection
   }

   return db
}

// elsewhere, as a helper - e.g. on the Workflow class
async function getDB(db, this.env) {
   if (!db) {
     db = // create a new connection
   }

   return db
}

SSakuraFC It works for sure but it also throws an exception for some reason. The only way ...

ddsgadget•5/6/25, 12:53 PM

Ah, ok, thanks very much! Will try that. Going to take some time to redo all our steps to add this everwhere. This I consider a bug, even if it was built this way that Steps can only return stringified JSON. The platform should handle that. And none of the examples show stringify and then parse, so how would anyone know to do that? https://developers.cloudflare.com/workflows/examples/send-invoices/ (we copied this example mostly for our workflows and use the {success:true} type of format.

Cloudflare Docs

Pay cart and send invoice

Send invoice when shopping cart is checked out and paid for

Austin B•5/6/25, 10:30 PM

I've recently built a solution on Workflows, and I'm trying to identity if I'm hitting some sort of Workers/Workflows limit or throttling. Maybe someone here can point me in the right direction.

High level summary of the solution: HTTP Worker request fires off 31 instances of one workflow. Each of those 31 fires off anywhere from 100-600 child workflows. Each of these child workflows makes a couple of Supabase calls.

Attached is a screenshot of the throughput of the worker bound to these workflows. No errors or retries occurred and the end results were what I expected. However, I don't understand the dramatic drop in throughput. Is there a way to see for sure if some service limit is being hit? (I'm on Workers Paid plan)

AAustin B I've recently built a solution on Workflows, and I'm trying to identity if I'm h...

Matt Silverlock•5/7/25, 12:57 AM

If you’re firing off 31 requests that each trigger hundreds of Workflows - you’re likely hitting rate limits: https://developers.cloudflare.com/workflows/reference/limits/

Cloudflare Docs

Limits

Limits that apply to authoring, deploying, and running Workflows are detailed below.

Matt Silverlock•5/7/25, 12:57 AM

(Sounds like a lot more than 450 per 10s)

Austin B•5/7/25, 2:46 AM

@Matt Silverlock I'm not sure which specific limit you're referring to, but as far as the workflow instance creation itself, I think I'm in the clear. I'm using

createBatch

createBatch

to do 100 at a time within each of the initial 31. The total amount gets queued up very quickly. The graph above is the steps processing inside those from my understanding.

I don't see anything about 450 per 10s in the link you mentioned. I've looked through the docs for Workers limits and Workflows limits, and nothing is standing out to me as obvious that my setup would be hitting. Admittedly, I'm not sure my understanding of subrequests and how steps within workflows translate to the workers limits is totally correct.

I was hoping to see some evidence within CloudFlare's dashboard itself as to what is being limited. But I can't find any errors or metrics indicating what limit might be getting hit.

AAustin B @Matt Silverlock I'm not sure which specific limit you're referring to, but as f...

Matt Silverlock•5/7/25, 2:50 AM

Sorry, I mean 100 / sec. 31 x 100 per request within a few seconds exceeds that? If not, can you expand on your creation rate? 31 requests over what period? Why 31?

Austin B•5/7/25, 2:54 AM

@Matt Silverlock 31 is arbitrary and data driven. Iterating over some entities and firing off workflows for related child entities. My understanding is that a single createBatch only counts as 1 request towards the rate limit. I'm getting this from these docs: https://developers.cloudflare.com/workflows/build/workers-api/#createbatch

So each of those 31 is not doing more than 7 createBatch calls (each batch with up to 100). Unless I'm misinterpreting the docs, this seems well within that limit. Also to be clear I'm not seeing issues with the workflow instances being created slowly. It is once they are actually queued, how quickly they get processed. The chart I showed above is not workflow instance creation

AAustin B @Matt Silverlock 31 is arbitrary and data driven. Iterating over some entities a...

Matt Silverlock•5/7/25, 3:02 AM

createBatch would limit that. Are you hitting all of this from a single IP? I’d open a Support ticket as hard to dive into over Discord.

We hit a new error message: `Workflow V2: Error processing batch 5 { "message": "Aborting engine:

Similar Threads

Similar Threads

Similar Threads