Handling Dynamic Links with Crawlee PlaywrightCrawler

I’m working on a project using PlaywrightCrawler to scrape links from a dynamic JavaScript-rendered website. The challenge is that the <a> tags don’t have href attributes, so I need to click on them and capture the resulting URLs. - Delayed Link Rendering: Links are dynamically rendered with JavaScript, often taking time due to a loader. How can I ensure all links are loaded before clicking? - Navigation Issues: Some links don’t navigate as expected or fail when trying to open in a new context. - Memory Overload: I get the warning "Memory is critically overloaded" during crawls I've attached images of my code (it was too long so I couldn't paste it) How can I handle these issues more efficiently, especially for dynamic and JavaScript-heavy sites? I would appreciate any help
No description
No description
3 Replies
Hall
Hall5mo ago
Someone will reply to you shortly. In the meantime, this might help:
Alexey Udovydchenko
First and foremost allocate more RAM, then, with enough memory, performance might be good enough to avoid navigation and loading issues
extended-salmon
extended-salmonOP5mo ago
Alright, thank you!

Did you find this page helpful?