CA
flat-fuchsia
How do I add headers to BasicCrawler?
I have a URL that requires some headers to be accessible, how do i add headers to BasicCrawler so I can crawl it?
2 Replies
typical-coral•3y ago
When you add request try request
headers property
https://crawlee.dev/api/core/class/Request#headersRequest | API | Crawlee
Represents a URL to be crawled, optionally including HTTP method, headers, payload and other metadata.
The
Request object also stores information about errors that occurred during processing of the request.
Each Request instance has the uniqueKey property, which can be either specified
manually in the constructor or generated automaticall...like-gold•3y ago
Just remember BasicCrawler doesn't do any requests by default, you need to use the provided
sendRequesthttps://crawlee.dev/api/basic-crawler/interface/BasicCrawlingContext#sendRequest
or just use CheerioCrawler