CA
Crawlee & Apify•13mo ago
flat-fuchsia

Selenium WebDriverWait() not available?

Hello, I'm running a python Actor on a page that has some dynamic content that loads slowly. I'm using WebDriverWait() in my code that works outside of apify. But when I run this code in apify I see this error message: 
2024-05-05T19:36:27.317Z ERROR Cannot extract data from https://slow.example.com/.
2024-05-05T19:36:27.320Z       Traceback (most recent call last):
2024-05-05T19:36:27.322Z         File "/usr/src/app/src/main.py", line 74, in main
2024-05-05T19:36:27.324Z           ad_elements = WebDriverWait(driver, 35).until(
2024-05-05T19:36:27.326Z                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-05T19:36:27.328Z         File "/usr/local/lib/python3.11/site-packages/selenium/webdriver/support/wait.py", line 95, in until
2024-05-05T19:36:27.329Z           raise TimeoutException(message, screen, stacktrace)
2024-05-05T19:36:27.331Z       selenium.common.exceptions.TimeoutException: Message:
2024-05-05T19:36:27.504Z INFO  Exiting actor ({"exit_code": 0})
2024-05-05T19:36:27.317Z ERROR Cannot extract data from https://slow.example.com/.
2024-05-05T19:36:27.320Z       Traceback (most recent call last):
2024-05-05T19:36:27.322Z         File "/usr/src/app/src/main.py", line 74, in main
2024-05-05T19:36:27.324Z           ad_elements = WebDriverWait(driver, 35).until(
2024-05-05T19:36:27.326Z                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-05T19:36:27.328Z         File "/usr/local/lib/python3.11/site-packages/selenium/webdriver/support/wait.py", line 95, in until
2024-05-05T19:36:27.329Z           raise TimeoutException(message, screen, stacktrace)
2024-05-05T19:36:27.331Z       selenium.common.exceptions.TimeoutException: Message:
2024-05-05T19:36:27.504Z INFO  Exiting actor ({"exit_code": 0})
I don't see anything in the docs for Python, but I do see references to context.waitFor() which looks like JS. Is this type of waiting only supported in JS?
1 Reply
flat-fuchsia
flat-fuchsiaOP•13mo ago
Just to follow up, I think this is a problem in selenium. Locally it doesn't seem to work in headless mode. I would up rewriting my function in js using puppeteer and that worked. 🤷

Did you find this page helpful?