How to tell PlaywrightCrawler to wait
In the playwrightUtils I have the option to tell Playwright to wait until content is loaded (This can be done by calling the "gotoExtended" function and providing "DirectNavigationOptions") or I can tell it to wait a certain amount of seconds for the content to load before exiting (This can be done in the "infiniteScroll" function by providing "InfiniteScrollOptions").
My question is, can this somehow also be done by using the main PlaywrightCrawler class? Just calling the "gotoExtended" or "infiniteScroll" does not seem to give me the option to use all other features that the PlaywrightCrawler provides, such as using a proxy server and so on.
2 Replies
generous-apricot•3y ago
If I understand correctly, you want to access the timeout and waitUntil options for a given crawl in the PlaywrightCrawler class. You need to add a preNavigationHooks entry and modify timeout and waitUntil to whatever values you prefer.
manual-pinkOP•3y ago
Thank you. That worked.