No Observability To What is Failed in creating workflow
Hey, I'm not sure if workflows is the right destination, but I feel this could be a more general issue.
I've encounter in a couple of times, that when I batch create (<4500) workflow instances but at 400 or something, I experienced significant wall time for the request to complete as queued build up, this contradict to what cloudflare mentioned in the docs of 4500 concurrent workflow instances. But also, when I have several hunderd of instances being queued, I got a ton of internal error that I have no clue of what's going on, and the queued instances often take a long time for them to get running. Sometimes, when looking at the log, I got no visibility as to what is preventing me from spawning more workflows.
So, is observability panel the right way to access those information because I've got no visibility as to what's going on with my request.
{
"source": {
"message": "internal error",
"exception": {
"stack": " at callFetcher (cloudflare-internal:workflows-api:24:15)\n at async WorkflowImpl.create (cloudflare-internal:workflows-api:78:24)\n at async handleScheduledEvent (index.js:77373:18)\n at async Object.scheduled (index.js:81123:5)",
"name": "Error",
"message": "internal error",
"timestamp": 1749823912326
},
"$cloudflare": {
"$metadata": {
"id": "01JXMT3YC6AXV8VV4G079ZTV9E",
"type": "cf-worker",
"error": "internal error"
}
}
},
"requestId": "3YWPOIJ1EJVZRW9Z"
},
"$metadata": {
"id": "01JXMT3YC6AXV8VV4G079ZTV9E",
"requestId": "3YWPOIJ1EJVZRW9Z",
"trigger": "* 13-21 * * mon-fri",
"service": "inmoney-api",
"level": "error",
"error": "internal error",
"message": "internal error",
"account": "1a846a3aa6bfc55b1f8ca8bee4c4bdce",
"type": "cf-worker",
"fingerprint": "1ef5291ce4c67da75b431843ca5a4c27",
"origin": "cron"
},
"links": []
}
8 Replies
Hey, you'd have better luck in #workflows , can just copy paste this in there. The workflows team/cf employees monitor that channel but not these
Hey Chaika, nobody is responding in the workflows thread. as well as the community.cloudflare.com
I figured that if I can find as what is really going on by having better visbility. I can get to know and fix the issue.
In my non-cloudflare employee opinion you might be overloading your account's workflow Durable Objects. They've blogged before that Workflow's architecture is all per user backed by Durable Objects: https://blog.cloudflare.com/building-workflows-durable-execution-on-workers/ . Durable Objects are single threaded v8 isolates, each one has a soft limit of ~1k rps but will be less if the code its running is more expensive, always going to be a hard ceiling
the team would have to comment more on if you're hitting anything known with inefficiencies
ok, that I may understand.
hmm I think there's been a few other reports of this, let me see
yes, I certainly looked into the past incident where someone from cloudflare suggest to batch create 40 at a time. Then I go significantly under.
hmm yea I was looking through old posts/messages and stumbled on this: https://discord.com/channels/595317990191398933/1296923707792560189/1345492333734662225
Understood! I'll read that article to better understand thank you @Chaika