CA
useful-bronze
Trying to get rid out of 429 errors
Hello, do you have some nice tips how to get rid out of 429? I am not exactly how parrarellism working here, but I am afraid, even I am putting in the sleep the process, the other parallel request as considered as request and they can lead to 429, is there any nice tip/best practice how can I defend against it ? 😄
2 Replies
View post on community site
This post has been pushed to the community knowledgebase. Any replies in this thread will be synced to the community site.
Apify Community
Hi @tomasj97
You can limit the maximum parallelism. And the number of tasks per minute max_concurrency=100, max_tasks_per_minute=10 - https://crawlee.dev/python/docs/guides/scaling-crawlers
Scaling crawlers | Crawlee for Python · Fast, reliable crawlers.
Crawlee helps you build and maintain your Python crawlers. It's open source and modern, with type hints for Python to help you catch bugs early.