What happens when there's an active workflow running and there's a new version deployed? My understa

Aajay | dreamlit.ai What happens when there's an active workflow running and there's a new version d...

M

Any running Workflow runs as-is today. Looking at options to provide config for this (e.g. terminate/restart all old instances on a deploy).

If it invokes another Workflow via a binding, that is still going to be the "latest" version. The old script isn't kept around. Workflows is no different here - if Worker A calls Worker B over a Service Binding, and you re-deploy Worker B, Worker A will always (only) call the latest code.

R

rodrigo•2/26/25, 9:19 PM

Running into (what I believe) are some rate limit errors when scheduling a workflow. I count around ~80k requests but only around 10k got scheduled/queued. I believe this error is related:

Error: Durable Object is overloaded. Too many requests queued.

Error: Durable Object is overloaded. Too many requests queued.

Error: Durable Object is overloaded. Too many requests queued.

Error: Durable Object is overloaded. Too many requests queued.

What's the rate limit for scheduling workflows? The

Maximum number of queued instances

Maximum number of queued instances

Maximum number of queued instances

Maximum number of queued instances looks like 100k per the docs

Rrichw Hi all. With workflows, what's the best practice for pausing for user input? Let...

D

DanB•2/26/25, 9:51 PM

Maybe have 2 separate workflows.
One for everything up to the email (including), and the other is triggered by the user responding to the email.

wf1 --> send email --> sleep (max_time_to_wait_for_user_to_respond) --> handle user not responding
wf2 --> cancel wf1 --> handle user email response

D

DanB•2/26/25, 10:09 PM

I run a workflow from a worker. The logs of the worker go to cloudflare logs. In the logs, I see an error line with a single word "run". This is happening on every execution of the workflow, and it is not coming from my code. The workflow itself runs correctly and completes all steps normally. Any clue?

DDanB I run a workflow from a worker. The logs of the worker go to cloudflare logs. In...

K

kchro3•2/26/25, 11:06 PM

i'm seeing this happening too, would love support / explanation on what these internal errors are

Nnoman yall, i'm trying to to use whisper to transcribe some audio as part of a workflo...

M

Matt Silverlock•2/26/25, 11:27 PM

How big is the file? You will likely need to stream it in / stream the response out as otherwise the file + transcription will have to remain buffered in memory, in full, the whole time.

Nnoman the `AI.run` method doesn't like receiving a `ReadableStream` for the audio, oth...

I

Isaac McFadyen•2/27/25, 12:56 AM

Unfortunately I don't believe you can stream using the AI binding, because internally it stringifies all inputs to JSON and that can't be streamed: https://github.com/cloudflare/workerd/blob/main/src/cloudflare/internal/ai-api.ts#L118

Rrodrigo Running into (what I believe) are some rate limit errors when scheduling a workf...

K

kchro3•2/27/25, 1:55 AM

hey, can we get support on this? seeing lots of failures with starting workflows from a worker

K

kchro3•2/27/25, 7:09 PM

the workflows dashboard is really slow to load... is this happening for anyone else?

K

kchro3•2/27/25, 7:22 PM

Seeing this error:

{
  "$workers": {
    "diagnosticsChannelEvents": [],
    "truncated": false,
    "event": {
      "rpcMethod": "run"
    },
    "scriptName": "...",
    "outcome": "exception",
    "eventType": "rpc",
    "entrypoint": "...",
    "scriptVersion": {
      "id": "..."
    },
    "executionModel": "stateless",
    "requestId": "...",
    "wallTimeMs": 30006,
    "cpuTimeMs": 0
  },
  "level": "error",
  "message": "run",
  "$metadata": {
    "now": "2025-02-27T19:19:13.214Z",
    "id": "...",
    "requestId": "...",
    "service": "...",
    "error": "run",
    "message": "run",
    "account": "...",
    "type": "cf-worker-event",
    "fingerprint": "...",
    "origin": "rpc"
  }

}

{
  "$workers": {
    "diagnosticsChannelEvents": [],
    "truncated": false,
    "event": {
      "rpcMethod": "run"
    },
    "scriptName": "...",
    "outcome": "exception",
    "eventType": "rpc",
    "entrypoint": "...",
    "scriptVersion": {
      "id": "..."
    },
    "executionModel": "stateless",
    "requestId": "...",
    "wallTimeMs": 30006,
    "cpuTimeMs": 0
  },
  "level": "error",
  "message": "run",
  "$metadata": {
    "now": "2025-02-27T19:19:13.214Z",
    "id": "...",
    "requestId": "...",
    "service": "...",
    "error": "run",
    "message": "run",
    "account": "...",
    "type": "cf-worker-event",
    "fingerprint": "...",
    "origin": "rpc"
  }

}

{
  "$workers": {
    "diagnosticsChannelEvents": [],
    "truncated": false,
    "event": {
      "rpcMethod": "run"
    },
    "scriptName": "...",
    "outcome": "exception",
    "eventType": "rpc",
    "entrypoint": "...",
    "scriptVersion": {
      "id": "..."
    },
    "executionModel": "stateless",
    "requestId": "...",
    "wallTimeMs": 30006,
    "cpuTimeMs": 0
  },
  "level": "error",
  "message": "run",
  "$metadata": {
    "now": "2025-02-27T19:19:13.214Z",
    "id": "...",
    "requestId": "...",
    "service": "...",
    "error": "run",
    "message": "run",
    "account": "...",
    "type": "cf-worker-event",
    "fingerprint": "...",
    "origin": "rpc"
  }

}

{
  "$workers": {
    "diagnosticsChannelEvents": [],
    "truncated": false,
    "event": {
      "rpcMethod": "run"
    },
    "scriptName": "...",
    "outcome": "exception",
    "eventType": "rpc",
    "entrypoint": "...",
    "scriptVersion": {
      "id": "..."
    },
    "executionModel": "stateless",
    "requestId": "...",
    "wallTimeMs": 30006,
    "cpuTimeMs": 0
  },
  "level": "error",
  "message": "run",
  "$metadata": {
    "now": "2025-02-27T19:19:13.214Z",
    "id": "...",
    "requestId": "...",
    "service": "...",
    "error": "run",
    "message": "run",
    "account": "...",
    "type": "cf-worker-event",
    "fingerprint": "...",
    "origin": "rpc"
  }

}

Can I get assistance with understanding what this means and where it's coming from?

J

Jikyu•2/27/25, 7:30 PM

Is there a way to stop new instances of a workflow from spawning through the dashboard / cli? I did something naughty and have created an infinite loop of workflows - basically each workflow would create another workflow, and they would resolve too fast for me to cancel it through the dashboard

K

kchro3•2/27/25, 7:45 PM

$ wrangler deploy --env stage

✘ [ERROR] A request to the Cloudflare API (/accounts/.../workflows/...-workflow-stage) failed.

workflows.api.error.internal_server [code: 10001]

seeing this issue now too...

A

Aslak Hellesoy•2/27/25, 11:58 PM

How do people test Cloudflare Workflows? I've created a little library that lets me do it more eaasily, but I wonder if there is another recommeded way to do it.

https://github.com/oselvar/noflare (sorry, the docs aren't that great yet - look at the example code if you're interested)

GitHub

GitHub - oselvar/noflare: Test helpers for Cloudflare Workers

Test helpers for Cloudflare Workers. Contribute to oselvar/noflare development by creating an account on GitHub.

K

kchro3•2/28/25, 1:41 AM

I dont think it’s supported by wrangler yet. I’ve had to hack around to get any vite tests to run at all.

I just put the logic in a different file that I can unit test, but I cant do integration tests or test workflows end to end.

A

Aslak Hellesoy•2/28/25, 7:54 AM

@kchro3 yeah I came to the same conclusion. Noflare lets me test workflows end to end, and it's been a huge help to us. Example: https://github.com/oselvar/noflare/blob/main/src/examples/CalculateCubeWorkflow.test.ts

GitHub

noflare/src/examples/CalculateCubeWorkflow.test.ts at main · oselva...

Test helpers for Cloudflare Workers. Contribute to oselvar/noflare development by creating an account on GitHub.

JJikyu Is there a way to stop new instances of a workflow from spawning through the das...

M

Matt Silverlock•2/28/25, 11:19 AM

Not yet. Maybe we need a

terminateAll

terminateAll

terminateAll

terminateAll here, although I would suggest redeploying the code that is spawning them to at least stop the initial run. cc @Seekerdasbatatas

Kkchro3 I dont think it’s supported by wrangler yet. I’ve had to hack around to get any ...

S

Seekerdasbatatas•2/28/25, 11:22 AM

I have a PR to get

vitest-pool-workers

vitest-pool-workers

vitest-pool-workers

vitest-pool-workers to not crash with Workflows: https://github.com/cloudflare/workers-sdk/pull/8208, hopefully gets merged really :soontm: - you should be able to do some e2e testing with this, but we have more utils to properly test steps in our roadmap.

MMatt Silverlock Not yet. Maybe we need a `terminateAll` here, although I would suggest redeployi...

J

Jikyu•2/28/25, 11:24 AM

I was able to catch one and cancel it, but will try that method next time

D

DanB•2/28/25, 5:15 PM

I have to run some migration process on ~2000 users. For each user I have to run a couple of requests and update some DB record.
The migration for each user is going to be a workflow.
My question is how should I loop through all users to execute the workflow?
Is it ok to run it from a worker? Can I start 2000 workflows from a single worker request, or is there some limit?
Or, should that loop be a workflow on its own? Read the list of users, then start a step for each user that is dynamically named after the userId?

Kkchro3 Seeing this error: ``` { "$workers": { "diagnosticsChannelEvents": [], ...

K

kchro3•2/28/25, 5:24 PM

still am looking for some help on this issue. we're seeing our worker fail when creating a workflow, and we're getting tons of logs for durable object is overloaded / failures that just say "run"

i think it may be a limitation on how fast you can trigger workflows, even if you are under the concurrent execution limit (4500) or if you are under the total queueable limit (100K)

Kkchro3 still am looking for some help on this issue. we're seeing our worker fail when ...

M

Matt Silverlock•3/1/25, 2:17 AM

Right now, yes. Still working through raw throughput improvements.

K

kchro3•3/1/25, 2:23 AM

ack, is there a rough QPS guideline for triggering workflows?

D

DanB•3/1/25, 7:32 PM

Hi @Matt - I think that cf internal errors should not leak to the workers logs. I took me a while to try and figure out what they mean, until I realized that there's nothing I could or should do about it. Moreover, it affects the error rate that I see in the log, potentially masking real errors that I have to be aware of.

D

DanB•3/1/25, 7:55 PM

If I have a workflow that is being called from a worker which is sending its logs to cf logs .
In the workflow code, I use console.log().
Where can I expect to see the workflow logs?

DDanB If I have a workflow that is being called from a worker which is sending its log...

M

Matt Silverlock•3/1/25, 8:23 PM

In the Worker’s logs.

Kkchro3 ack, is there a rough QPS guideline for triggering workflows?

M

Matt Silverlock•3/1/25, 8:26 PM

Ballpark 100-250/sec as a very conservative number. Creating Workflows is the biggest “cost” right now - once they are running it’s much easier to scale out. Still optimizing.

MMatt Silverlock Ballpark 100-250/sec as a very conservative number. Creating Workflows is the bi...

K

kchro3•3/1/25, 8:29 PM

Ok, that explains a lot! Thank you

F

fatwang2•3/2/25, 10:13 AM

Hey, everyone, I want to share my open-source repo, GitPush, which is an AI GitHub release tracker powered by workflows and workers AI and Email Routing: https://github.com/fatwang2/gitpush

GitHub

GitHub - fatwang2/gitpush: an AI Github Rlease Tracker Powered by C...

an AI Github Rlease Tracker Powered by Cloudflare. Contribute to fatwang2/gitpush development by creating an account on GitHub.

F

fatwang2•3/2/25, 10:15 AM

cloudflare full stack

U

usf•3/3/25, 3:34 PM

Hi, is there any Workflows downtime right now? i've been seeing a lot of 500 internal server errors today, the status page doesn't mention it.

Uusf Hi, is there any Workflows downtime right now? i've been seeing a lot of 500 int...

S

Seekerdasbatatas•3/3/25, 3:57 PM

not as far as I'm aware

, could you give me your account id so I can investigate further?

SSeekerdasbatatas not as far as I'm aware 😅 , could you give me your account id so I can investig...

U

usf•3/3/25, 3:58 PM

sure here's it

adbf3651ede24f0e60bc37f80673f51f

adbf3651ede24f0e60bc37f80673f51f

adbf3651ede24f0e60bc37f80673f51f

adbf3651ede24f0e60bc37f80673f51f, Thanks!

B

Boragorn•3/3/25, 7:02 PM

Do the workflows have a hard timeout defined? I'm running a request (lambda invoke) inside the workflow that can take up to 3 minutes to complete, but the steps fail after approximately 90 seconds.

B

Boragorn•3/3/25, 7:07 PM

btw -- 524 error

B

Boragorn•3/4/25, 2:13 PM

I haven't tried with websockt, tried with a few different request handlers from aws-sdk, node fetch and xhr, tried turning on the keep-alive and it didn't work. I'll give it a shot with websockets though

U

usf•3/4/25, 2:20 PM

Hey, i'm noticing a lot of errors with the messages thrown by step.do. and workflows not loading on cloudflare since yesterday, this wasn't happening before

A call to blockConcurrencyWhile() in a Durable Object waited for too long. The call was canceled and the Durable Object was reset

A call to blockConcurrencyWhile() in a Durable Object waited for too long. The call was canceled and the Durable Object was reset

A call to blockConcurrencyWhile() in a Durable Object waited for too long. The call was canceled and the Durable Object was reset

A call to blockConcurrencyWhile() in a Durable Object waited for too long. The call was canceled and the Durable Object was reset

is it just me or is anyone else having the same issue?

U

usf•3/4/25, 2:23 PM

S

Sameer Ali•3/4/25, 5:52 PM

It might be naive question but i am having hard time seeing console output. i have a workflow that makes call to AWS lambda and for some reasons lambda is returning error, to debug, i have added logs, which do not appear in workflow dashboard, and i see nothing in wrangler tail. is there any guide or recommendation on how to debug the workflow?

S

Saqib•3/4/25, 8:26 PM

I am trying to use supabase inside cloudflare workflows, and I am also using @microlabs/otel-cf-workers package. It looks like they don't work well together.

{
  message: "TypeError: Cannot read properties of undefined (reading 'fetch')",
  details: "TypeError: Cannot read properties of undefined (reading 'fetch')\n" +
    '    at null.<anonymous> (file:///****/node_modules/.pnpm/@microlabs+otel-cf-workers@1.0.0-rc.40/node_modules/@microlabs/otel-cf-workers/src/instrumentation/fetch.ts:243:80)\n' +
    '    at Object.apply (file:///****/node_modules/.pnpm/@microlabs+otel-cf-workers@1.0.0-rc.40/node_modules/@microlabs/otel-cf-workers/src/instrumentation/fetch.ts:211:19)\n' +
    '    at proxyHandler.apply (file:///****/node_modules/.pnpm/@microlabs+otel-cf-workers@1.0.0-rc.40/node_modules/@microlabs/otel-cf-workers/src/wrap.ts:33:19)\n' +
    '    at fetch (file:///****/node_modules/.pnpm/@supabase+supabase-js@2.48.1/node_modules/@supabase/supabase-js/src/lib/fetch.ts:15:42)\n' +
    '    at null.<anonymous> (file:///****/node_modules/.pnpm/@supabase+supabase-js@2.48.1/node_modules/@supabase/supabase-js/src/lib/fetch.ts:46:12)\n' +
    '    at Generator.next (<anonymous>)\n' +
    '    at fulfilled (file:///****/.wrangler/tmp/dev-QyRdTX/index.js:78978:24)',
  hint: '',
  code: ''
}

{
  message: "TypeError: Cannot read properties of undefined (reading 'fetch')",
  details: "TypeError: Cannot read properties of undefined (reading 'fetch')\n" +
    '    at null.<anonymous> (file:///****/node_modules/.pnpm/@microlabs+otel-cf-workers@1.0.0-rc.40/node_modules/@microlabs/otel-cf-workers/src/instrumentation/fetch.ts:243:80)\n' +
    '    at Object.apply (file:///****/node_modules/.pnpm/@microlabs+otel-cf-workers@1.0.0-rc.40/node_modules/@microlabs/otel-cf-workers/src/instrumentation/fetch.ts:211:19)\n' +
    '    at proxyHandler.apply (file:///****/node_modules/.pnpm/@microlabs+otel-cf-workers@1.0.0-rc.40/node_modules/@microlabs/otel-cf-workers/src/wrap.ts:33:19)\n' +
    '    at fetch (file:///****/node_modules/.pnpm/@supabase+supabase-js@2.48.1/node_modules/@supabase/supabase-js/src/lib/fetch.ts:15:42)\n' +
    '    at null.<anonymous> (file:///****/node_modules/.pnpm/@supabase+supabase-js@2.48.1/node_modules/@supabase/supabase-js/src/lib/fetch.ts:46:12)\n' +
    '    at Generator.next (<anonymous>)\n' +
    '    at fulfilled (file:///****/.wrangler/tmp/dev-QyRdTX/index.js:78978:24)',
  hint: '',
  code: ''
}

{
  message: "TypeError: Cannot read properties of undefined (reading 'fetch')",
  details: "TypeError: Cannot read properties of undefined (reading 'fetch')\n" +
    '    at null.<anonymous> (file:///****/node_modules/.pnpm/@microlabs+otel-cf-workers@1.0.0-rc.40/node_modules/@microlabs/otel-cf-workers/src/instrumentation/fetch.ts:243:80)\n' +
    '    at Object.apply (file:///****/node_modules/.pnpm/@microlabs+otel-cf-workers@1.0.0-rc.40/node_modules/@microlabs/otel-cf-workers/src/instrumentation/fetch.ts:211:19)\n' +
    '    at proxyHandler.apply (file:///****/node_modules/.pnpm/@microlabs+otel-cf-workers@1.0.0-rc.40/node_modules/@microlabs/otel-cf-workers/src/wrap.ts:33:19)\n' +
    '    at fetch (file:///****/node_modules/.pnpm/@supabase+supabase-js@2.48.1/node_modules/@supabase/supabase-js/src/lib/fetch.ts:15:42)\n' +
    '    at null.<anonymous> (file:///****/node_modules/.pnpm/@supabase+supabase-js@2.48.1/node_modules/@supabase/supabase-js/src/lib/fetch.ts:46:12)\n' +
    '    at Generator.next (<anonymous>)\n' +
    '    at fulfilled (file:///****/.wrangler/tmp/dev-QyRdTX/index.js:78978:24)',
  hint: '',
  code: ''
}

{
  message: "TypeError: Cannot read properties of undefined (reading 'fetch')",
  details: "TypeError: Cannot read properties of undefined (reading 'fetch')\n" +
    '    at null.<anonymous> (file:///****/node_modules/.pnpm/@microlabs+otel-cf-workers@1.0.0-rc.40/node_modules/@microlabs/otel-cf-workers/src/instrumentation/fetch.ts:243:80)\n' +
    '    at Object.apply (file:///****/node_modules/.pnpm/@microlabs+otel-cf-workers@1.0.0-rc.40/node_modules/@microlabs/otel-cf-workers/src/instrumentation/fetch.ts:211:19)\n' +
    '    at proxyHandler.apply (file:///****/node_modules/.pnpm/@microlabs+otel-cf-workers@1.0.0-rc.40/node_modules/@microlabs/otel-cf-workers/src/wrap.ts:33:19)\n' +
    '    at fetch (file:///****/node_modules/.pnpm/@supabase+supabase-js@2.48.1/node_modules/@supabase/supabase-js/src/lib/fetch.ts:15:42)\n' +
    '    at null.<anonymous> (file:///****/node_modules/.pnpm/@supabase+supabase-js@2.48.1/node_modules/@supabase/supabase-js/src/lib/fetch.ts:46:12)\n' +
    '    at Generator.next (<anonymous>)\n' +
    '    at fulfilled (file:///****/.wrangler/tmp/dev-QyRdTX/index.js:78978:24)',
  hint: '',
  code: ''
}

If I remove the @microlabs/otel-cf-workers, it works fine. How can I fix it?

MMatt Silverlock Ballpark 100-250/sec as a very conservative number. Creating Workflows is the bi...

K

kchro3•3/5/25, 2:49 PM

Just a follow-up here, can this be documented somewhere, especially if there are plans to increase this throughput, and do you have recommendations on how to deal with this?

for example, should we have retries when creating the stub or when awaiting the create function?

Original message was deleted

K

kchro3•3/5/25, 4:33 PM

workflows are fire-and-forget, so the last step of the workflow can be to a database/kv store/durable object

What happens when there's an active workflow running and there's a new version deployed? My understa

Similar Threads

Similar Threads

Similar Threads