CA
deep-jade
Limiting Number of Concurrent Agents?
Hi -- I'm just starting to work with the platform. I've been very impressed so far but I'm running into a challenge constraining things so I don't get a lot of what are effectively "out of resources" errors.
What I am doing is trying to use Apify as the crawler end of an airtable knowlege tracking tool we use internally. Users submit links into an Airtable, we pull the links into a cannon including content with Zapier as the orchestration tool. Apify is much better at the pulling in part, but the challenge I'm running into is we get a lot of errors because this is a bursty process and we request too many page parsings at the same time. Zapier doesn't seem to have a way to do this elegantly but I'm hoping Apify does.
Ideally there is a setting somewhere that says "only run up to 4 concurrent agents at once for this task" or "queue agents by available RAM on the account." Is there any way to approach this, especially from a no-code solution angle?
2 Replies
like-gold•2y ago
Hey, limiting the number of concurrent Actor runs or queuing them is unfortunately not available at the moment. What about running the Actor for larger batches in the input.
deep-jadeOP•2y ago
A very belated thanks for getting back to me. I agree running batches is ideal but the input comes one at a time. Most of the times it is slow so it does not matter but sometimes we get bursts that cause said failure. But will ponder how one could re-think things to feed batches.