got many 429 status code when crawled the target site,even though proxies. How to optimise my code?

Hi,guys. I am a python coder but not good at nodejs. I make a crawler to bulk check the infomation by the crawlee. This is my option : useSessionPool: true, useIncognitoPages: true, and I using the Residential Proxies . But I found that the ip of some pages as same as the others. I want launch diffrent browser context in each target url . But I don't know how to do that. Anybody could help me ? Sorry for my poor English . Thanks
No description
2 Replies
NeoNomade
NeoNomade2y ago
https://crawlee.dev/api/core/interface/SessionOptions Set maxErrorScore to 1 and a new proxy from your proxy config will be assigned after each error
absent-sapphire
absent-sapphireOP2y ago
Thanks for your help. I will test your method.

Did you find this page helpful?