How to enqueue links until a certain "depth"
I want to crawl until a certain depth of the website. Is there an option on enqueueLinks or somewhere else for this?
3 Replies
vicious-gold•2y ago
updated version
you can save the current
depth in userData of each request, increase it in the request handler when enqueuing new links and then stop when reaching the desired depthRequest | API | Crawlee
Represents a URL to be crawled, optionally including HTTP method, headers, payload and other metadata.
The
Request object also stores information about errors that occurred during processing of the request.
Each Request instance has the uniqueKey property, which can be either specified
manually in the constructor or generated automaticall...genetic-orange•2y ago
limit is just limiting number of request enqueued
ratty-blushOP•2y ago
Thx guys!