429
Hello π
Does the HTTP crawler use fingerprinting by default when sending requests?
Because I get 429 too many requests from the API I am web scraping, and I suspect that my header values are the cause of it.
13 Replies
Yes they are enabled by default. I believe 429 are because of IP address. Try to use proxies
xenial-blackβ’4mo ago
I prefer residential rotating proxies
sunny-greenβ’4mo ago
When the request is sent and logged into the request_queues folder the only header i see is
"headers": {
"Content-Type": "application/json",
},
Have you tried proxies?
sunny-greenβ’4mo ago
not yet
@fierDeToiMonGrand just advanced to level 1! Thanks for your contributions! π
sunny-greenβ’4mo ago
tried the proxy and it works well thanks !
Great. I'd start with data centers since they're lot cheaper than residential and would probably help with IPs rotation (but if they're antibots is more sophisticated then residential would yield better results.
Anyway, glad it helped! Feel free to mark this as solved βΊοΈ
sunny-greenβ’4mo ago
how can i mark it resolved?
I just did β
Have a nice day/evening/night βΊοΈ
sunny-greenβ’4mo ago
thanks! You too π
like-goldβ’4mo ago
Yes, you can add a check to switch the proxy when the request status code is 429, which will help resolve the issue.
@ΠΠ°Π½\ just advanced to level 2! Thanks for your contributions! π