Crawlee Not scraping when provided with the same link twice
I'm using a REST API to access my Scraper which will be provided with a
url
. And, it will scrape the URL and return me the Information.
Now, When the Server is first started, and the request is made. It works correctly. It scrapes the data and responds with the results. But when another request is made with the same URL as the parameter, It doesn't scrape the URL and instead just says that All requests from the queue have been processed, the crawler will shut down
. And shuts down. It needs to rescrape the Data and return it as my App needs it to work.
I'm using the PlayWright Crawler. the code is attached below.
I think it has something to do with the request crawler. But, I'm not sure. Can anyone help? Thanks!

2 Replies
@ThalfPant just advanced to level 1! Thanks for your contributions! 🎉
automatic-azure•2y ago
It may be helpful if you could share the code here.