I am not sure what has happened with
I am not sure what has happened with workflows product recently, but since yesterday, the product just doesn't work for me, i've had to stop everything, and now looking at alternative solutions.
I can't trigger / pause / resume / terminate any workflows, l met with the error code
10001 as earlier.
Over 3000 workflows failed yesterday with the exact same error This instance experienced an internal error which prevented it from running. If this is persistent, please contact support: https://cfl.re/3WgEyrH
Any new workflow l try to start this morning goes straight into queued state and will never run.
I have 4K+ messages going into dead-letter-queues because workflows can't be instantiated.
@James / @Diogo Ferreira any help would be appreciated, as this product was working perfectly up until yesterday moring.38 Replies
Untouchable instances, some are 2 days+ old
Even the cf dashboard cannot fetch information about workflows:
waited some time, and tried to trigger a few hundred instances, again either inactive or stuck running, i am a bit shocked how this is product has been broken for over 24hrs now
@Thomas Ankcorn i am not sure if you could help on any of the above?
Thanks Ollie, flagging with the team
Hi Ollie, sorry for the issues you are seeing. Since yesterday, we have started monitoring your account, and we have initiated a cleanup process. This was the cause of the 3000 workflow instances failed, as during the cleanup process, the system purges irrecoverable instances.
Regarding the new instances created this morning, they should have run normally, which doesn't see to be the case.
We are looking into it as we speak.
Ah amazing, and thanks a lot, sorry to sound like am complaining, l do love the cloudflare products and everything has been working so smoothly, i've ran almost 1 million workflow instances, so real credit to the platform being able to handle all that
Let me also point out, that the issue is limited to your account.
Good to know
Not that it is of any help to you, I know, but just for the sack of clarification
Deffo good to know, otherwise one can start to wonder if there is a wide spread issue, and then just constantly spamming the status page
Also some new instances that i've been triggering seem to be at least running, not sure if all of them are but some of them are, there just seems to be a backlog of stuck "queued" ones
Ollie, does this issue still persist for you?
Not always, so say if l try to load the page, maybe 2/5 fail to load, but this happens mainly on workflow heavy pages, e.g. one of the workflows that has ran 800,000+ times
so l wonder if its calculating the stats per page load or not?
Yes, that can affect it
@Ollie I can see that for the last hour, your Workflows are being completed. Yes, there are some instances still in running state (we are looking into those), but newer instances are running correctly.
Can you please confirm?
Yep this looks to be correct
I can see from my external logging thrings are running but now l can no longer see via the cf dashboard:
That's strange, I can query your account with no issues
Yeah really weird, still not able to see anything on it, tried on a different PC as well, nothing
Same on my mobile, just checked
All with the same
internal_server ?Weirdly no, so from the network l can see the request: https://dash.cloudflare.com/api/v4/accounts/0601cdee659d3394e0d37db62e73705c/workflows?per_page=25&page=1&name=undefined is being sucessful
Hummm
Yeah really weird, and also sometimes it never even makes that network request when reloading
Yeah all of the other pages work fine except for any pages under workflows
On reloads l seem to get a flash of the page structure loading then it vanishes
Can you check now, please?
am not sure what magic you did but that works
SO l can see its cleared up a lot now, only have around 65 instances stuck in a running state now
Yep, I can also see that. We are trying to understand what is causing that
thanks a lot for the help so far btw
I believe you are facing a race condition here
im putting the workflows through their paces then 😛
Hey, I'm facing the same issue. I think it's related to the Cloudflare dashboard—maybe it was recently updated to a new version, and older versions cached in users' browsers aren't able to fetch the latest version correctly. It could also be a conflict between the old and new versions of the dashboard. Everything was working fine until yesterday, so it's possible that some changes made by the Cloudflare frontend team in the last 24 hours triggered this issue.
That said, if you open the dashboard in an incognito tab, everything works fine—probably because incognito loads the latest version without using the cached files.
this is not related to only workflows but may be related to the frontend of cloudflare
Yes, we had a regression with a Front-end release, which was already reverted. Clearing your cache will fix it.
Yes after doing force reload ( ctrl + shift + R ) it fixed
@Ollie you have stop creating/running new instances, correct?
Yeah l didn't want to interfer with your debugging so just stopped all
You can continue
Ah perfect will do
All new instances of workflows seem to be running pretty smoothly, though am keeping numbers low, should l start to ramp up to see if the race condition gets hit again?
Oh dear @Diogo Ferreira we are back to the same issue after having ramped up:
{ code: 10001, message: "workflows.api.error.internal_server" }Instances are starting to pile up and get stuck:
CF Dashboards not loading anymore for me either:
