Basic Crawlee how do I use my own proxies?
'Did not expect property
proxyConfiguration to exist, got [object Object] in object BasicCrawlerOptions'2 Replies
mute-gold•2y ago
Hello @drowning in milk, the BasicCrawler does not handle proxy configuration at all, that's why it's constructor options do not expect this field. With BasicCrawler, you have to implement the fetching logic yourself (using any library you want).
conscious-sapphireOP•2y ago
Yep, makes sense. I forgot I was using gotscraping. But I wonder, what is the job of BasicCrawler then?
Just queueing the requests?
Also, I found a problem with
addRequestsBatched. The crawling session will terminate before all the requests are added. The workaround was to change the batchsize to 10k. I have >50k requests