got many 429 status code when crawled the target site,even though proxies. How to optimise my code?
Hi,guys. I am a python coder but not good at nodejs. I make a crawler to bulk check the infomation by the crawlee.
This is my option :
useSessionPool: true,
useIncognitoPages: true,
and I using the Residential Proxies .
But I found that the ip of some pages as same as the others.
I want launch diffrent browser context in each target url . But I don't know how to do that.
Anybody could help me ?
Sorry for my poor English .
Thanks

2 Replies
https://crawlee.dev/api/core/interface/SessionOptions
Set maxErrorScore to 1 and a new proxy from your proxy config will be assigned after each error
absent-sapphireOP•2y ago
Thanks for your help. I will test your method.