BasicCrawler request information
Hi, when using BasicCrawler, how do you get info about Hostname, IP Address, User Agent, AS Number for the request, i need to provide them to our client to be able to give us access to be able to scrape
1 Reply
correct-apricot•2y ago
hey, you could try to print out the request object in the request handler and inspect the information, there are some default values, but in BasicCrawler, you generally set up all options and configuration (such as proxies) by yourself. To get proxy info, you can print out the proxyConfiguration object
Proxy Management | Crawlee
Using proxies to get around those annoying IP-blocks