CA
Crawlee & Apify•3y ago
rare-sapphire

Can someone tell me what this error means? It says browser closed unexpectedly but not as to why.

2023-02-28T14:33:25.456Z ACTOR: Pulling Docker image from repository.
2023-02-28T14:33:26.293Z ACTOR: Creating Docker container.
2023-02-28T14:33:26.614Z ACTOR: Starting Docker container.
2023-02-28T14:34:04.313Z Traceback (most recent call last):
2023-02-28T14:34:04.314Z   File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
2023-02-28T14:34:04.315Z     return _run_code(code, main_globals, None,
2023-02-28T14:34:04.315Z   File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
2023-02-28T14:34:04.316Z     exec(code, run_globals)
2023-02-28T14:34:04.316Z   File "/usr/src/app/src/__main__.py", line 3, in <module>
2023-02-28T14:34:04.317Z     main()
2023-02-28T14:34:04.317Z   File "/usr/src/app/src/main.py", line 112, in main
2023-02-28T14:34:04.317Z     asyncio.get_event_loop().run_until_complete(run())
2023-02-28T14:34:04.318Z   File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
2023-02-28T14:34:04.319Z     return future.result()
2023-02-28T14:34:04.319Z   File "/usr/src/app/src/main.py", line 30, in run
2023-02-28T14:34:04.320Z     browser = await launch(options=options)
2023-02-28T14:34:04.320Z   File "/opt/venv/lib/python3.9/site-packages/pyppeteer/launcher.py", line 307, in launch
2023-02-28T14:34:04.320Z     return await Launcher(options, **kwargs).launch()
2023-02-28T14:34:04.321Z   File "/opt/venv/lib/python3.9/site-packages/pyppeteer/launcher.py", line 168, in launch
2023-02-28T14:34:04.321Z     self.browserWSEndpoint = get_ws_endpoint(self.url)
2023-02-28T14:34:04.322Z   File "/opt/venv/lib/python3.9/site-packages/pyppeteer/launcher.py", line 227, in get_ws_endpoint
2023-02-28T14:34:04.322Z     raise BrowserError('Browser closed unexpectedly:\n')
2023-02-28T14:34:04.323Z pyppeteer.errors.BrowserError: Browser closed unexpectedly:
2023-02-28T14:33:25.456Z ACTOR: Pulling Docker image from repository.
2023-02-28T14:33:26.293Z ACTOR: Creating Docker container.
2023-02-28T14:33:26.614Z ACTOR: Starting Docker container.
2023-02-28T14:34:04.313Z Traceback (most recent call last):
2023-02-28T14:34:04.314Z   File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
2023-02-28T14:34:04.315Z     return _run_code(code, main_globals, None,
2023-02-28T14:34:04.315Z   File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
2023-02-28T14:34:04.316Z     exec(code, run_globals)
2023-02-28T14:34:04.316Z   File "/usr/src/app/src/__main__.py", line 3, in <module>
2023-02-28T14:34:04.317Z     main()
2023-02-28T14:34:04.317Z   File "/usr/src/app/src/main.py", line 112, in main
2023-02-28T14:34:04.317Z     asyncio.get_event_loop().run_until_complete(run())
2023-02-28T14:34:04.318Z   File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
2023-02-28T14:34:04.319Z     return future.result()
2023-02-28T14:34:04.319Z   File "/usr/src/app/src/main.py", line 30, in run
2023-02-28T14:34:04.320Z     browser = await launch(options=options)
2023-02-28T14:34:04.320Z   File "/opt/venv/lib/python3.9/site-packages/pyppeteer/launcher.py", line 307, in launch
2023-02-28T14:34:04.320Z     return await Launcher(options, **kwargs).launch()
2023-02-28T14:34:04.321Z   File "/opt/venv/lib/python3.9/site-packages/pyppeteer/launcher.py", line 168, in launch
2023-02-28T14:34:04.321Z     self.browserWSEndpoint = get_ws_endpoint(self.url)
2023-02-28T14:34:04.322Z   File "/opt/venv/lib/python3.9/site-packages/pyppeteer/launcher.py", line 227, in get_ws_endpoint
2023-02-28T14:34:04.322Z     raise BrowserError('Browser closed unexpectedly:\n')
2023-02-28T14:34:04.323Z pyppeteer.errors.BrowserError: Browser closed unexpectedly:
13 Replies
passive-yellow
passive-yellow•3y ago
Are you running the code on Apify platform? Can you share the run id? Are you using the beta Python SDK template? There’s no browser in the default Docker image. So even if you install the pyppeteer, the Dockerfile is probably wrong and missing a lot of dependencies....
rare-sapphire
rare-sapphireOP•3y ago
im using the regular python, not the beta sdk here's the run id 6VKUYfzm58AaGWni5
passive-yellow
passive-yellow•3y ago
Thanks, we are checking it So yes - my answer above is correct. There's no browser in the image.
rare-sapphire
rare-sapphireOP•3y ago
what's that mean? i can't use python for browser scraping? or is there a way to add a browser to the image?
passive-yellow
passive-yellow•3y ago
You can, but you will have to take care of browser download and other necessary dependencies. The executable is just not bundled in this image.
rare-sapphire
rare-sapphireOP•3y ago
the logs show it downloaded chromium i thought. is there something else i have to do for it?
passive-yellow
passive-yellow•3y ago
I don't think so - it downloaded pyppeteer, but not the browser executable: https://api.apify.com/v2/logs/ESecEJo1QfYsuhRBe. I haven't personally worked with custom docker images, so I would sadly won't be able to further advance you here, but it's missing the browser and maybe some other deps...
rare-sapphire
rare-sapphireOP•3y ago
what is this then 
2023-02-28T22:23:38.638Z [INFO] Starting Chromium download.
2023-02-28T22:23:39.154Z 
  0% 0.00/109M [00:00<?, ?b/s]
 21% 22.3M/109M [00:00<00:00, 223Mb/s]
 43% 46.8M/109M [00:00<00:00, 236Mb/s]
 65% 70.4M/109M [00:00<00:00, 236Mb/s]
 86% 94.1M/109M [00:00<00:00, 236Mb/s]
100% 109M/109M [00:00<00:00, 240Mb/s]
2023-02-28T22:23:39.155Z [INFO] Beginning extraction
2023-02-28T22:23:41.224Z [INFO] Chromium extracted to: /root/.local/share/pyppeteer/local-chromium/588429
2023-02-28T22:23:38.638Z [INFO] Starting Chromium download.
2023-02-28T22:23:39.154Z 
  0% 0.00/109M [00:00<?, ?b/s]
 21% 22.3M/109M [00:00<00:00, 223Mb/s]
 43% 46.8M/109M [00:00<00:00, 236Mb/s]
 65% 70.4M/109M [00:00<00:00, 236Mb/s]
 86% 94.1M/109M [00:00<00:00, 236Mb/s]
100% 109M/109M [00:00<00:00, 240Mb/s]
2023-02-28T22:23:39.155Z [INFO] Beginning extraction
2023-02-28T22:23:41.224Z [INFO] Chromium extracted to: /root/.local/share/pyppeteer/local-chromium/588429
passive-yellow
passive-yellow•3y ago
It's log from the run, not from the build though - I could guess that it's downloaded during the run, and pyppeteer is not really aware of it/not aware where it's extracted
rare-sapphire
rare-sapphireOP•3y ago
oh, do you think adding an option that points to it would work?
passive-yellow
passive-yellow•3y ago
That might work, but that's still quite weird that in downloads during the run itself. Here's what I got from the team: 
It’s most likely missing some dependencies and system libraries. The Node.js images have some complicated dependency installations to make sure the browser has everything it needs. When pyppeteer downloads Chromium, it does not install all the deps
It’s most likely missing some dependencies and system libraries. The Node.js images have some complicated dependency installations to make sure the browser has everything it needs. When pyppeteer downloads Chromium, it does not install all the deps
rare-sapphire
rare-sapphireOP•3y ago
ah, that makes sense then. i'll try using actual puppeteer then, although i dont know any js 😅
passive-yellow
passive-yellow•3y ago
Depending on the use-case - you could try our generic scrapers - https://apify.com/apify/web-scraper or https://apify.com/apify/playwright-scraper - I'm pretty sure you should pick it up - it does not requires much coding, mainly specifying the selectors, etc
Apify
Web Scraper · Apify
Crawls arbitrary websites using the Chrome browser and extracts data from pages using a provided JavaScript code. The actor supports both recursive crawling and lists of URLs and automatically manages concurrency for maximum performance. This is Apify's basic tool for web crawling and scraping.
Apify
Playwright Scraper · Apify
Crawls websites with the headless Chromium, Chrome, or Firefox browser and Playwright library using a provided server-side Node.js code. Supports both recursive crawling and a list of URLs. Supports login to a website.

Did you find this page helpful?