Crawlee & Apify•4mo ago

clean way to stop "request queue seems to be stuck for 300.0"

A scraper that I am developing, scrapes a SPA with infinite scrolling. This works fine, but after 300 seconds, I get a WARN , which spawns another playwright instance. This probably happens since I only handle 1 request (I do not add anything to the RequestQueue), from which I just have a while until finished condition is met.

[crawlee.storages._request_queue] WARN  The request queue seems to be stuck for 300.0s, resetting internal state. ({"queue_head_ids_pending": 0, "in_progress": ["tEyKIytjmqjtRvA"]})

[crawlee.storages._request_queue] WARN  The request queue seems to be stuck for 300.0s, resetting internal state. ({"queue_head_ids_pending": 0, "in_progress": ["tEyKIytjmqjtRvA"]})

What is a clean way to stop this from happening?

4 Replies

Hall•4mo ago

Someone will reply to you shortly. In the meantime, this might help: -# This post was marked as solved by DuxSec. View answer.

other-emerald•4mo ago

Hi, since the default timeout is 300 seconds (5 minutes), you can increase the timeout to allow more time for infinite scrolling.

rival-blackOP•4mo ago

Thank you ! For others:

Actor.config.internal_timeout = timedelta(seconds=xxx)

Actor.config.internal_timeout = timedelta(seconds=xxx)

MEE6•4mo ago

@DuxSec just advanced to level 1! Thanks for your contributions! 🎉

Gaming

Programming

clean way to stop "request queue seems to be stuck for 300.0"

Did you find this page helpful?