CA
Crawlee & Apify•3y ago
adverse-sapphire

PlaywrightCrawler.requestHandler: Error: mouse.move: Target page, context or browser has been closed

In the PlaywrightCrawler.requestHandler I calling page.mouse.move and sometimes I get this error: mouse.move: Target page, context or browser has been closed Here the sequence of calls: 
async requestHandler( {request, response, page, enqueueLinks, log, proxyInfo} )
{
    ...
    await sleep( interval );
    await page.mouse.move( rnd(100,400), rnd(40,300) );
    await sleep( interval );
    ...
    content = await page.content();
}
async requestHandler( {request, response, page, enqueueLinks, log, proxyInfo} )
{
    ...
    await sleep( interval );
    await page.mouse.move( rnd(100,400), rnd(40,300) );
    await sleep( interval );
    ...
    content = await page.content();
}
In case I catch the exception thrown in page.mouse.move and contine - than I get almost the same thing when calling page.content(): page.content: Target page, context or browser has been closed I would like to move mouse randomly - I think, the make my scraper "human-like". But something is going wrong here and I can not figure out what. Sometimes this code works and sometimes I see these errors! Pls help! UPDATE: and sometimes the error message is: ERROR requestHandler: Request failed and reached maximum retries. page.goto: Navigation failed because page was closed! I think, somehow these error messages are related...
4 Replies
ratty-blush
ratty-blush•3y ago
Page might detect bot and close itself, as a quick solution try Playwright (its less detectable)
foreign-sapphire
foreign-sapphire•3y ago
page close usually means it just crashed because your CPU or memory was overloaded. Try to decrease concurrency or increase system resources
adverse-sapphire
adverse-sapphireOP•3y ago
I do not think this is my case (minConcurrency=maxConcurrency=4, i do not think this is a lot) but I check this again.
CPU or memory was overloaded.
with CPU - this is clear. with RAM - do you think this is "not enough RAM for +1 browser" or "not enough RAM for the node process"? What is the best way to detect the shortage of RAM ?
foreign-sapphire
foreign-sapphire•3y ago
I mean the browser, it depends a lot on how big the page is. Just try to reduce concurrency to 1 and see if you still see the crashes. Also, don't set minConcurrency, because it cannot downscale then, use desiredConcurrency of autoscaledPoolOptions

Did you find this page helpful?