hello, we're heavy users of Workflows
hello, we're heavy users of Workflows and our system is hitting up against the 100 per 10 seconds "Maximum Workflow instance creation rate" limit. we are seeing the number of queued workflow instances as we are adding users.
we've submitted the Limit Increase Request Form twice and wanted to see if someone could help with increasing our limit there
10 Replies
Hey super late for me but tomorrow morning I will talk to the rest of team
Are you hitting limits fairly frequently?
Or is it just to prevent future problems?
we were hitting up against the limit (we have about 115 workflows constantly running) and we were seeing the number of queued instances going up to about 10-12.
actually, we realized that we could re-configure our jobs to basically run a bit longer (While staying under the 1024 step limit). and in doing so, the chances of overlapping workflows being instantiated went down. so that brought our queued instances back down.
these workflows are polling for us.
but we'd love some headroom on this as we scale up. the number of workflows is increasing linearly for us daily as we add users.
I am not sure I understand what the problem is.
You have 100+ workflows that should always be running but they end up roughly at the same time so they get recreated at the same time causing you to hit creation rate limit?
Or is it something else?
(I did not see a lot of 429 for your account)
fair! when i first reached out, our suspicion was around the creation rate but that looks to have been incorrect.
we can reach out when we actually near the creation rate. to your point, i don't think we're being affected by that metric currently
hey just coming back to this.
we're growing pretty quick here (now at 157 concurrent workflows) and would love if we could increase this limit to have some more headroom as our userbase grows.
we're concerned we will hit this limit shortly.
1. due to what you mentioned above where more of our workflows start at the same time
2. if we do more bursty actions that might spike our usage of workflows.
btw, is the creation limit per workflow?
It’s per account, is using queues to buffer creation an option or having a bit of jitter? (If not will chat with the team tomorrow)
ah i see, it's per account. then yeah it'd be great to get this increased. the 157+ is only for one workflow, but we also have several other workflows that are less predictable.
actually, we're leveraging Workflows for somewhat time-sensitive things (transaction email notification platform). so queueing is not ideal for our use case.
we do plan to implement batching/queueing for some of our time-insensitive actions though.
For context... we are an email notification service, the limit means at the moment we can only send 100 emails per 10 seconds 😬 in the case of bursty usage, this means we would almost certainly hit the limit
Just an FYI docs update as not landed yet but the creation rate limit is now 100 workflow creations per second
Unknown User•3w ago
Message Not Public
Sign In & Join Server To View
awesome! 100 per second (just to double confirm, 100 per second not per ten seconds) (edit: just saw the docs were updated and see the 100 per second) is definitely better!
and yes Mia! happy to hop on a call to talk about our use case. ideally sometime next Tues-Thurs (we are located on Eastern Time)
andrew@dreamlit.ai ajay@dreamlit.ai