Stop `keepAlive` crawler after all requests are finished
Hello,
I have two crawlers, one running with
keepAlive: true (call it A) and the second running normally (call it B), which adds request to the crawler A. After the cralwer B finishes I'd like to keep the crawler A running until it finishes all the requests and then stop the script. I tried the teardown method but it stops the crawler without finishing the queue.2 Replies
Hello Richie,
Took a quick look and I see that the
keepAlive: true option should only set the isFinishedFunction to false . ( https://github.com/apify/crawlee/pull/1452/files#diff-3606a49356303bfa2bfed957bc81701e3d45d4cdafd4d5182685ef826c93b63fR509-R511 )
You should be able to set your own isFinishedFunction function through the first crawler configuration https://sdk.apify.com/docs/2.3/typedefs/autoscaled-pool-options#isfinishedfunction so it may return true in case the second actor finished and the queue is empty.
I can not found the documentation for Crawlee/v3.0 but it still should be there with the same interface.criminal-purpleOP•3y ago
Ok, thanks I'll play with it