Download Delay
does crawlee support download delay? Like in Scrapy? Because I want to crawl a website but this website has delay before to load its content, so my current crawlee project didn't get the content of the website.
5 Replies
other-emerald•2y ago
hey, how about waiting for a selector?
metropolitan-bronzeOP•2y ago
I will scrape many and different website's so that why I'm not depending on a selector
maxRequestsPerMinute
optimistic-gold•2y ago
Use waitUntil() (https://crawlee.dev/api/playwright-crawler/namespace/playwrightUtils#waitUntil) Just find proper event to wait for.
Or You can use sleep() (https://crawlee.dev/api/utils/function/sleep) (not the best approach).
metropolitan-bronzeOP•2y ago
thanks for this, I tried both of these and set a usecase for each. Also I try the networkIdle