Cloudflare Developers•7mo ago

Does anyone else seeing some steps get stuck in running for quite some time?

VVlad Lykhonis Does anyone else seeing some steps get stuck in running for quite some time?

avenceslau•6/5/25, 3:54 PM

Can you please run /link/link so that we can have a look?

Nathan Scheele•6/5/25, 5:05 PM

Is there anyway to disable event logging (see below) when a workflow receives an Event after calling

step.waitForEvent

step.waitForEvent

? Not sure if it's happening in production, but it makes going through logs locally pretty messy.

{
  payload: {
    result: ... // my result data
  },
  type: 'sub-request-complete',
  timestamp: 2025-06-05T17:01:09.650Z
}

{
  payload: {
    result: ... // my result data
  },
  type: 'sub-request-complete',
  timestamp: 2025-06-05T17:01:09.650Z
}

Aavenceslau Can you please run `/link` so that we can have a look?

Vlad LykhonisOP•6/5/25, 5:39 PM

link?

Rapid•6/5/25, 8:49 PM

Lets say I have 5 workers that each one has a different job and I want to run them by a specific order, sometimes 1 - 2 - 3 sometimes 4 - 2 - 1, is there a way to run a workflow dynamically based on the choosen jobs?

Delong•6/6/25, 2:41 PM

Hello, I need help with the build. I constantly get this error during the build. And the error is very general, I don't know what to do to fix it. Can someone help?

10:38:31.885 ✘ [ERROR] A request to the Cloudflare API (/accounts/[account_id]/workflows/set-etf-profiles-workflow) failed.
10:38:31.885
10:38:31.885 workflows.api.error.internal_server [code: 10001]
10:38:31.885
10:38:31.885 If you think this is a bug, please open an issue at: https://github.com/cloudflare/workers-sdk/issues/new/choose

GitHub

Build software better, together

GitHub is where people build software. More than 150 million people use GitHub to discover, fork, and contribute to over 420 million projects.

Delong•6/6/25, 3:22 PM

and some of the workflow instance stuck at running, it does not proceed forward, it doesn't have steps been run. And while the limits page side there can be 4500 concurrent workflow runs at the same time. (Note I do have workers paid) I see my limit at 125.

Jjuanferreras any inherit incompatibility between Workers Rate Limiting e.g. `ratelimit` and a...

Troob•6/6/25, 3:45 PM

I have this same issue, were you able to find a solution?

Delong•6/6/25, 3:59 PM

Hey @Matt Silverlock , I saw your announcement of increasing to 4,500 concurrent workflow run within each account for user that have workers paid, but from the dashboard and what I am seeing, it never reaches 4,500, it get limited at ~120.

Delong•6/6/25, 4:15 PM

So I was observing workflows dashboard this morning, because I made a quite significant changes to my worker last night.

I was expecting approximately 400 instances per minute that are spawned from a parent workflow that are batched into 4 groups, 100 per 15 seconds. I was expecting all to be ran on their given interval. But instead of all been completed, I see a huge number of workflow instances being queued; then it accumulated at least 1 thousand of instances. I observed it on time, and turned off the scheduler 'cron'. Hence stopping the ability to generate more queued instances. I am not sure if there is anything from worker that is limiting the ability from achieving the ideal concurrent instances, or any other reason, please help

Also the workflow dashboard also been extremly slow. And whenever I push any new debugging changes, it failed the build

koraytaylan•6/6/25, 9:02 PM

what is the best option for having a static egress ip address for making some http calls via the fetch api?

Alec•6/7/25, 1:47 AM

Is there any way to debug a Attempt failed due to internal workflows errorAttempt failed due to internal workflows error error?

scook•6/7/25, 4:28 AM

Is using
waitUntil
waitUntil
supported / recommended in Workflows?
What is actually happening behind-the-scenes if I use it within a step? Does it count against the duration / compute time of that step? Will the step itself end or wait for the background process to complete? What if the workflow performs retries on a failed step -- what happens to the background process?

Sscook **Is using `waitUntil` supported / recommended in Workflows? ** What is actually...

Matt Silverlock•6/7/25, 9:43 AM

Firstly: what are you trying to achieve? (it will count against your compute time - it’s not a way to get free compute!)

Oreki•6/7/25, 1:29 PM

When i try to publish a workflow with workers i get this error: Token Permissions: If scopes are missing, you may need to logout and re-login.

Jehu•6/7/25, 2:55 PM

Does anyone have a workaround for testing workflows. See ticket below. Any help is much appreciated.

https://github.com/cloudflare/workers-sdk/issues/7414

GitHub

🐛 BUG: Vitest does not work with workflows · Issue #7414 · clo...

Which Cloudflare product(s) does this pertain to? Miniflare, Other What version(s) of the tool(s) are you using? 0.5.32 [@cloudflare/vitest-pool-workers], 3.91.0 [wrangler] What version of Node are...

OOreki When i try to publish a workflow with workers i get this error: Token Permission...

koraytaylan•6/8/25, 12:53 PM

this also happens to me time to time. retrying a couple more times self-corrects the issue. no need to mention i am certain that the token i use has all the necessary scopes to begin with in my case.

RRapid Lets say I have 5 workers that each one has a different job and I want to run th...

Rapid•6/8/25, 1:53 PM

Anyone?

Kkoraytaylan this also happens to me time to time. retrying a couple more times self-corrects...

Oreki•6/8/25, 3:17 PM

Yeah quite amazing, you just have to run the command multiple times and hope it works this time lol

RRapid Lets say I have 5 workers that each one has a different job and I want to run th...

koraytaylan•6/8/25, 7:04 PM

workflow steps are already dynamically runnable. so, at runtime, you can pick and chose which step should come next based on the workflow params or even by individual step outputs.

JJehu Does anyone have a workaround for testing workflows. See ticket below. Any help ...

koraytaylan•6/8/25, 7:08 PM

i did not really like the way that vitest plugin worked so i sticked with jest by sacrificing some functionality. i try to mock everything and try not to rely on anything but the logic i am testing so it makes sense to go this way in my case.

Kkoraytaylan workflow steps are already dynamically runnable. so, at runtime, you can pick an...

Rapid•6/9/25, 1:32 PM

Can you maybe guide me through that? Like link me something that explains on how to do it

sheffield•6/9/25, 9:52 PM

i randomly started getting this error when trying to create a workflow batch. it's been working fine up until now, but not sure where to go from here:

Converting circular structure to JSON
    --> starting at object with constructor 'PgTable'
    |     property 'id' -> object with constructor 'PgUUID'
    --- property 'table' closes the circle

Converting circular structure to JSON
    --> starting at object with constructor 'PgTable'
    |     property 'id' -> object with constructor 'PgUUID'
    --- property 'table' closes the circle

the params i'm passing to my workflow create batch don't contain the tabletable property anywhere, and i've verified the types are correct. it works fine locally—any ideas on how to root cause this?

sheffield•6/9/25, 10:05 PM

nevermind, fixed the issue—one of the parameters i was passing wasn't serializable

Nimesh•6/9/25, 11:39 PM

Hi, we am using both workflows and queues in our application. We noticed that workflows requires isolatedStorageisolatedStorage flag to be set to

false

false

(otherwise, vitest doesn't recognize the entrypoint?). However, since setting isolatedStorageisolatedStorage to

false

false

, we see a different error related to queues.

QueuesError [ERR_MULTIPLE_CONSUMERS]: Multiple consumers defined for queue "<queue-name>"

QueuesError [ERR_MULTIPLE_CONSUMERS]: Multiple consumers defined for queue "<queue-name>"

Any idea how we can solve this?

lent20254•6/10/25, 2:01 PM

I deleted a User API Key but it is still working.

lent20254•6/10/25, 2:01 PM

Surely this isnt expected?

lent20254•6/10/25, 2:04 PM

Is there a way to get comeone from CF to help?

lent20254•6/10/25, 2:45 PM

@avenceslau I can confirm the key works but see no keys in the UI

lent20254•6/10/25, 2:50 PM

Checking with this script I dont see any tokens in either:

Common headers

headers = {
"X-Auth-Email": CF_EMAIL,
"X-Auth-Key": CF_API_KEY,
"Content-Type": "application/json",
}

── 1. List all user-level API tokens ──

user_tokens_url = "https://api.cloudflare.com/client/v4/user/tokens"
resp = requests.get(user_tokens_url, headers=headers)
print("User-level tokens response:")
print(json.dumps(resp.json(), indent=2))

── 2. List all account-level API tokens ──

account_tokens_url = f"https://api.cloudflare.com/client/v4/accounts/{CF_ACCOUNT_ID}/tokens"
resp = requests.get(account_tokens_url, headers=headers)
print("\nAccount-level tokens response:")
print(json.dumps(resp.json(), indent=2))

Llent20254 I deleted a User API Key but it is still working.

Brandon•6/12/25, 1:22 PM

Hey lent — I'll look into this and let you know what I find out.

Victor•6/12/25, 5:50 PM

In the past ~2 hours I'm seeing an uptick in working binding

.create()

.create()

with the whole error being just

internal error

internal error

VVictor In the past ~2 hours I'm seeing an uptick in working binding `.create()` with th...

Victor•6/12/25, 5:54 PM

Also the dashboard is erroring out with 500500s intermittently with 1000110001: workflows.api.error.internal_serverworkflows.api.error.internal_server

ddsgadget•6/12/25, 6:08 PM

Is cloudflare down? B/c all our KV storage is erroring and the dashboard no longer works?

Chaika•6/12/25, 6:12 PM

CDN is fine (Discord is through it!), but there's def something messed with kv, known issue

ddsgadget•6/12/25, 6:13 PM

cloudflare dashboard doesn't work at all. i guess it uses KV on the backend. Anyway, always suprising to me that nothing is ever reported on the Cloudflare status page till way later, but something is up. how do you even inform Cloudflare that something is wrong?

koraytaylan•6/12/25, 6:28 PM

dashboard does not load at all. indefinitely verifying if i am a human.

Dddsgadget cloudflare dashboard doesn't work at all. i guess it uses KV on the backend. Any...

Chaika•6/12/25, 6:36 PM

They have tons of internal alerting so they noticed this super quickly, but if it's more limited in scope us (community champs) can escalate things

VVictor In the past ~2 hours I'm seeing an uptick in working binding `.create()` with th...

Victor•6/12/25, 10:26 PM

following up now that the outage has been resolved, the incident was marked as resolved at 21:31 UTC (the left most red column on the graph). I still got that group of 10, but also 20 more

internal error

internal error

on workflow binding

.create()

.create()

since. However, falling back to the HTTP workflow

POST .../{workflow_name}/instances

POST .../{workflow_name}/instances

api (with the exact same workflow id + params, it succeeds

Kkoraytaylan in workflow instance details page, output gets truncated for a particular "step ...

koraytaylan•6/13/25, 5:55 AM

i am still suffering from this.

Delong•6/13/25, 1:59 PM

Any plans to improve workflow dashboard experience, it is quite slow for no apparent reasons.

JJehu Does anyone have a workaround for testing workflows. See ticket below. Any help ...

Advany•6/13/25, 6:15 PM

where you able to solve this?

AAdvany where you able to solve this?

Jehu•6/14/25, 2:49 AM

Nope, we are in the CF purgatory when it comes to vitest + workflows. :NotLikeThis:

Dan (Openside)•6/14/25, 3:38 PM

When architecting workflows and steps, what are the pros/cons of using a queue in conjunction with a workflow vs. just running a loop and performing a step. Example: I read from the db a list of URLs that need to be moved over to R2. What would be better architecture: Read in the list, loop through and perform batch steps for each? Or add each one to the queue which then triggers a new workflow for each. Here is what I have come up with, curious to hear what else I should be thinking about.

No Queue (stay within workflow) approach:
Pros - avoid queue pricing; can aggregate results of all of them and then do more processing once they are all done.
Cons - may hit the 1024 steps limit depending on size of list and number of inner steps; all share same workflow storage so may hit 1GB storage limit

Use Queue (run new workflow per item) approach:
Pros - can rely on queue retries; starting new workflow reduces number of inner steps to avoid 1024 limit; allow for 1GB storage for each
Cons - consumes more workflow requests, increasing price; not able to easily process after all files in list are processed. added cost due to using queue.

Anything I'm missing or misinterpreting? Curious how others architect your workflows. thanks

JJehu Nope, we are in the CF purgatory when it comes to vitest + workflows. :NotLikeTh...

Advany•6/14/25, 6:01 PM

to bad... wont use workflows untill this is sovled

scook•6/15/25, 1:49 AM

Has anyone run into issues locally when invoking Workflows from async
waitUntil
waitUntil
code?
I have a Worker that attempts to trigger a Workflow like so:

// Worker
ctx.waitUntil(
  nonBlockingFunc({ ... })
);

async nonBlockingFunc() {
  const workflowInstance = await env.MY_WORKFLOW.create();
  // await new Promise((resolve) => setTimeout(resolve, 2000)); // sleep fixes???
  return workflowInstance.id;
}

// Worker
ctx.waitUntil(
  nonBlockingFunc({ ... })
);

async nonBlockingFunc() {
  const workflowInstance = await env.MY_WORKFLOW.create();
  // await new Promise((resolve) => setTimeout(resolve, 2000)); // sleep fixes???
  return workflowInstance.id;
}

On my local dev, it appears to successfully create the Workflow instance (I can see the instance id), but the Workflow itself is never actually run. However, if I insert a sleep after the invocation, then the Workflow does actually get run. Afaict this is only happening on my local and not in production.

My friendly ChatGPT claims that this is a shortcoming of Cloudflare local development / Miniflare not fully respecting the async tasks in the

waitUntil

waitUntil

, because it garbage collects the state and tears down the execution environment after the Workflow instance is registered but BEFORE the workflow is actually run. The sleep prevents the premature shutdown of the Worker execution context.

Can anyone verify if this is true?

IvoDev•6/15/25, 11:49 PM

Can i use durable objects inside workflows and viceversa?

Jehu•6/16/25, 12:02 AM

Yes, bindings all the way

IvoDev•6/16/25, 12:04 AM

amazing ! ultimate durability

alex (he/him)•6/16/25, 7:10 AM

Can I put everything that doesn't need an external service directly inside the

run

run

method without wrapping it inside a

step.do

step.do

. My examples are

zod

zod

validation of params or something like

const ai = new GoogleGenAI()

const ai = new GoogleGenAI()

.. I theory I don't want to persist those, I'm fine if they run again on a new try. So would it be best to put them in the global

run

run

scope?

ivanovm•6/16/25, 11:39 PM

hey cloudflare team, i love the platform otherwise but it is genuinely unacceptable to have 524 timeout on outgoing requests at 100 seconds. almost every major ai model can take over 100 seconds to respond, cloudflare becomes unusable by default for ai applications

Does anyone else seeing some steps get stuck in running for quite some time?

Common headers

── 1. List all user-level API tokens ──

── 2. List all account-level API tokens ──

Similar Threads

Similar Threads

Common headers

── 1. List all user-level API tokens ──

── 2. List all account-level API tokens ──

Similar Threads