I have been messing with Workflows for a week or so for ETL and the limits feel incredibly hard to d

I have been messing with Workflows for a week or so for ETL and the limits feel incredibly hard to deal with, specifically the 1k sub requests. Sure I could implement some complex job splitting, but it just feels like a lot of juggling and a really rough DevEx. For example, let's say you have a JSON file with a few thousand records in it you want to import into your D1 instance you're going to hit the limits almost immediately and this isn't really a "lot of work" for production.
Am I missing something here? It seems to basically only really be a viable tool for trickle emails and other short, simple workflows.

OndreianOP•11/17/25, 4:06 PM

fan-out, fan-in doesn't really seem to be supported because if you start scheduling sub-workflows you will just hit the limit then as well in your orchestrator.

OOndreian I have been messing with Workflows for a week or so for ETL and the limits feel ...

xvvvyz•11/17/25, 4:16 PM

my solution to this was to pass the work off to a container.

Xxvvvyz my solution to this was to pass the work off to a container.

OndreianOP•11/17/25, 4:23 PM

When you say "the work" do you mean basically every

step.do

step.do

call or just some sort of top-level orchestration that would kick off smaller workflows? If you mean all of

step.do

step.do

calls, what value is wrapping it in a workflow even adding then since you lose all of the Workflow APIs inside of the Container where the processing is being done?

xvvvyz•11/17/25, 4:26 PM

only step in my workflow has too many sub requests.

xvvvyz•11/17/25, 4:27 PM

actually now that I think of it do container requests count towards sub requests?

OndreianOP•11/17/25, 4:33 PM

I don't think the rate limiter counts them as a tree, but as direct subrequests from the Workflow

xvvvyz•11/17/25, 4:37 PM

anyway this is specifically what I do on the step that would hit the limit

await step.do("generate images", async () => {
  void containerFetch(this.env, "/pdf-to-images", {
    document_id: document.id,
    file: pdf,
    workflow_id: event.instanceId,
  });
});

if (!hasLastPageImage) {
  await step.waitForEvent("images generated", {
    timeout: "1 hour",
    type: WorkflowEventType.ImagesGenerated,
  });
}

await step.do("generate images", async () => {
  void containerFetch(this.env, "/pdf-to-images", {
    document_id: document.id,
    file: pdf,
    workflow_id: event.instanceId,
  });
});

if (!hasLastPageImage) {
  await step.waitForEvent("images generated", {
    timeout: "1 hour",
    type: WorkflowEventType.ImagesGenerated,
  });
}

xvvvyz•11/17/25, 4:38 PM

since it has to upload images one by one to r2

OndreianOP•11/17/25, 4:52 PM

I guess that's a viable work around. If I am evaluating that though, it's still weirdly complex and developer hostile to require that for kind of basic "workflow" operations. I would have expected that was something you could easily handle in the Workflow declaratively. The whole point of a Workflow is to let you break down complex tasks into simple, repeatable processes, not create a lot of odd architectural dependencies. What happens if you re-deploy the container in the middle of a long workflow run? Feels like it allows some weirdly potentially, non-deterministic designs to become common place. Basically now you'll need some sort of mutex deployment for your Container to make sure you don't break any in-flight work with that, or I guess a transaction system.

xvvvyz•11/17/25, 5:00 PM

you're not wrong!

xvvvyz•11/17/25, 5:01 PM

don't think it's the best but at a certain point I need to ship something (:

OndreianOP•11/17/25, 5:08 PM

Not a critique of your workaround at all, as you said, you needed to ship something. Just kind of an odd choice for Cloudflare to design Workflows in a manner that doesn't really let you get too far without pulling in a lot of complexity. I guess I will just go back to my basic Queue processor fan-out strategy, it felt weird because of the lack of observability but as you've kindly advised, I don't really seem to have any other option

OndreianOP•11/17/25, 5:08 PM

i very much appreciate that you ran into the same problem and had to come up with a similar sort of workaround for it, that assures me I am not just missing some glaringly obvious thing in the docs

James•11/17/25, 8:45 PM

I find workflows not great if you can't work well within its bounds. I ended up creating my own fan-out/fan-in task mgmt system w queues and a simple workflow type tracking table. A lot more flexibility for a couple days work in building an orchestrator of my own.
For mass job creation, I had to do something similar and build out a meta-handler for managing sub requests. Once I had that working, I just left it in the background and the queue mgmt is handled. I guess it's not the ideal DX, but at the same time, I prefer local control over beta-magic.

OOndreian I guess that's a viable work around. If I am evaluating that though, it's still ...

1984 Ford Laser•11/17/25, 8:55 PM

Queues might be more suited to what you're trying to do?

ByteBeast•11/18/25, 6:24 AM

do anyone have any idea about after I execute sleep until for long period of time, how much time cloudflare takes to be in hibernation mode?

OOndreian I have been messing with Workflows for a week or so for ETL and the limits feel ...

Deltachaos•11/18/25, 2:23 PM

Yes this is something i have experienced as well. I am working around this by having abstracted away the actual workflow stuff and now I am building "steps" in away that they can ether be messages in a queue or newly dispatched workflows as both get fresh limits. I dont know exactly how the steps stuff is implemented, but in my opinion the limits should apply on a per step basis, instead of on a per workflow basis

kleinpetr•11/18/25, 8:07 PM

Hi I would like to understand the lifecycle of workflow, I am trying to replace cron by workflow, so I've done something like

run(){
while(true){
  await step.do(...)

  await step.sleep(5 min)
}
}

run(){
while(true){
  await step.do(...)

  await step.sleep(5 min)
}
}

locally it seems to work, but on remote I am getting internal error without any message or log. So I am wondering if it's possible to do it that way.

also I am experiencing another internal error while I am trying to restart instance. So I am not sure if it depends on my approach or workflows are just like beta..

kleinpetr•11/18/25, 10:44 PM

I am trying to end up workflow with custom error, so I used NonRetryableError NonRetryableError outside of step, but the the message was not used as an error.

Aiden•11/18/25, 11:09 PM

Is there a way to redact values from the step output, so that it doesn't appear in the Workflow logs (workflows/example-workflow/instance/runIDworkflows/example-workflow/instance/runID)? We have a step that passes API keys to another step, and it's all logged in the workflow runs... which isn't ideal.

kleinpetr•11/21/25, 9:10 AM

Just to be sure — there is no way to restart() only a specific step in a workflow, right?

I'm asking because I'm thinking about idempotency. For example, in the official example for sending an invoice after payment creation (https://developers.cloudflare.com/workflows/examples/send-invoices/)

If the email step fails, the entire workflow ends up in an errored state. If I still want to deliver that email, it seems like there’s no option to retry just that failed step.

Within a standard Queue, this would normally end up in a DLQ, and I could re-post all failed email events later.

Is there any equivalent approach in Workflows? I’m trying to understand the right mental model for handling retries or partial failures. Thanks

Paul•11/21/25, 3:36 PM

I am able to use email catch all, but sending seems to be problematic, just want to verify is email SENDING different from receiving in terms of permissions, like does the sending require a private beta? If so, how can I request access? Thanks

PPaul I am able to use email catch all, but sending seems to be problematic, just want...

Paul•11/21/25, 3:53 PM

Also if anyone can share specific valid example how to configure a worker to have a EMAIL binding in toml config file, I've tried like 20 different patterns, can't figure it out, which leads me to wonder if there is a private beta somewhere?

I keep seeing in my cloudlfare logs:

"logs": [
{
"message": [
"EMAIL binding is not configured in wrangler.toml"
],
"level": "error",
"timestamp": 1763740274632
}

PPaul Also if anyone can share specific valid example how to configure a worker to hav...

Paul•11/21/25, 4:05 PM

I found the interest Google Form, submitted.

PPaul Also if anyone can share specific valid example how to configure a worker to hav...

kleinpetr•11/21/25, 5:02 PM

not sure if it's related to #workflows

Kkleinpetr not sure if it's related to #workflows

Paul•11/21/25, 5:10 PM

Ya me either

I had to pick a channel, and figured the way this blog talked about the feature actually shows the sending of emails from within Workers. https://blog.cloudflare.com/email-service/

The Cloudflare Blog

Announcing Cloudflare Email Service’s private beta

Today, we’re launching Cloudflare Email Service. Send and receive email directly from your Workers with native bindings—no API keys needed. We're unifying email sending and routing into a single service built for developers. Sign up for the private beta.

kleinpetr•11/22/25, 5:35 PM

the

timeout

timeout

within

waitForEvent

waitForEvent

means that after that timeout workflow continues or workflow is being stopped?

Kkleinpetr the `timeout` within `waitForEvent` means that after that timeout workflow conti...

kleinpetr•11/22/25, 6:53 PM

and also what happen when I would send event before the waiting for event starts?

Kkleinpetr and also what happen when I would send event before the waiting for event starts...

recruit•11/22/25, 7:43 PM

The timeout inside waitForEvent is basically a stopwatch, not a kill switch.

When your workflow hits waitForEvent, it pauses and listens for the event.
If the timeout expires before the event arrives, the workflow simply continues past that point (usually via the timeout branch or default continuation).
The workflow does NOT stop, crash, or end unless you coded it to end at that point.
Events that arrive before waitForEvent starts do not count and are not buffered, unless the workflow engine explicitly supports event history or replay queues

Quantum64•11/22/25, 9:43 PM

how exactly does this make sense

QQuantum64 how exactly does this make sense 😂

Isaac McFadyen•11/23/25, 1:53 AM

Do you mean CPU time being more than wall time? 1s of CPU time is counted as 1 core being busy for 1 second, so theoretically if you were using 54/21 = 2.57 cores of CPU at 100% you could get those numbers.

Isaac McFadyen•11/23/25, 1:53 AM

Oh wait, I didn't even notice the minutes vs seconds bit. That's definitely... interesting :lul:

IIsaac McFadyen Do you mean CPU time being more than wall time? 1s of CPU time is counted as 1 c...

Quantum64•11/23/25, 2:13 AM

plus workers can use a most a single cpu core right, webworkers or parallelism in general isn't supported

Quantum64•11/23/25, 2:13 AM

but yeah I have used way more than 21 seconds of wall time

Quantum64•11/23/25, 2:14 AM

at least like 100 times more

Isaac McFadyen•11/23/25, 2:14 AM

In JS-land yes, I'm not sure whether any of the functions that call into

workerd

workerd

's C++ are multithreaded or not though (i.e. hashing)

Quantum64•11/23/25, 2:15 AM

hm, fair

kleinpetr•11/23/25, 7:52 AM

guys, there's probably a bug in new sdk, especially with error type, since I re-declared the workflow instance types by my own I am getting typecheck error, so when I open sdk the error is now typed just as a string

Kkleinpetr guys, there's probably a bug in new sdk, especially with error type, since I re-...

kleinpetr•11/23/25, 8:20 AM

it's very interesting, because until now I used error.message and I am 100% sure the message is there, because I have this implementation on FE, but if I see to worker-types codebase, it says that this type was made 1 year ago..

Rrecruit The timeout inside waitForEvent is basically a stopwatch, not a kill switch. Wh...

avenceslau•11/23/25, 12:35 PM

(@kleinpetr tagging you since you asked) This is mostly incorrect, when the waitForEvent times out the workflow errors (you can wrap the wait for event in a trycatch if you want your workflow to proceed nonetheless). You can send the event before the waitForEvent begins, as long as the instance has been created.

Aavenceslau (@kleinpetr tagging you since you asked) This is mostly incorrect, when the wait...

recruit•11/23/25, 2:05 PM

Oh my apologies, I don't have that much experience.

Aavenceslau (@kleinpetr tagging you since you asked) This is mostly incorrect, when the wait...

kleinpetr•11/24/25, 12:26 PM

thanks for clarify

Nathan Scheele•11/24/25, 4:38 PM

Is it possible to artificially limit the number of concurrent workflow instances, similar to the max_concurrencymax_concurrency setting for consumer Queues?

1984 Ford Laser•11/24/25, 10:13 PM

Is it possible to have multiple waitForEvent()waitForEvent()s running at the same time for a single Workflow instance? I am starting a Workflow when a user begins uploading a series of files to R2, and using R2 Event Notifications to notify the Workflow when each file is complete. I don't think I can guarantee the order of upload so I would prefer to wait for all files to be uploaded at the same time. Otherwise I will use a single

waitForEvent

waitForEvent

that will re-trigger another wait..wait.. until all required files are uploaded

Maulik Tyagi•11/25/25, 10:03 AM

are there any good resources to learn in detail about workflows apart from docs? Would be really helpful to get some resources to see into

wesbos•11/25/25, 10:26 PM

What is the blessed way to keep track of instances? The binding doesn't have a way to list instances, and I found the HTTP API, but that is prod only?

I want to make a dashboard like Vercel's Workflows https://x.com/wesbos/status/1993415964143370302

1984 Ford Laser•11/25/25, 11:19 PM

Not sure if this is a bug or not, but having a period in a

waitForEvent

waitForEvent

type parameter, it breaks things. So like, video.mp4video.mp4 doesn't work, giving workflow.invalid_event_type, even though they're both strings. Does the period need to be escaped?

kleinpetr•11/26/25, 10:12 PM

I've started to get interesting error time to time, would you have any idea?

ByteBeast•11/28/25, 11:41 AM

Hello everyone, I have use cloudflare workflow in my app, now I want to keep my server's database sync with workflow instance what are the approach for it? does cloudflare provides any webhook using which I can sync my workflow with my database? I want to store status and steps of cloudflare workflow in my db.

1984 Ford Laser•11/29/25, 10:42 PM

I'd love to have something similar to Actors/Agents Schedule API where I can have a Workflow peridically poll something to check for status, basically an opposite of

waitForEvent

waitForEvent

. I know I can defer this polling to a DO, but would be useful to do it within the existing flow

I have been messing with Workflows for a week or so for ETL and the limits feel incredibly hard to d

Similar Threads