Selenium + Chrome Instagram Scraper cannot find the Search button when I run it in Apfiy..
Hey everyone,
I have built an Instagram Scraper using Selenium and Chrome that works perfectly until I deploy it as an actor here on Apify.
It signs in fine but fails every time no matter what I do or try when it gets to the Search button.
I have iterated through:
1) search_icon = WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.CSS_SELECTOR, "svg[aria-label='Search']"))
)
search_icon.click()
-----
2) search_icon = WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.XPATH, "//span[contains(., 'Search')]"))
)
search_icon.click()
-----
3) search_icon = WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.XPATH, "//svg[@aria-label='Search']"))
)
search_icon.click()
----
4) try:
search_button = WebDriverWait(driver, 30).until(
EC.element_to_be_clickable((
By.XPATH,
"//a[.//svg[@aria-label='Search'] and .//span[normalize-space()='Search']]"
))
)
# Scroll the element into view just in case
driver.execute_script("arguments[0].scrollIntoView(true);", search_button)
search_button.click()
except TimeoutException:
print("Search button not clickable.")
----
5) search_button = WebDriverWait(driver, 30).until(
EC.element_to_be_clickable((
By.XPATH,
"//a[.//svg[@aria-label='Search'] and .//span[normalize-space()='Search']]"
))
)
driver.execute_script("arguments[0].scrollIntoView(true);", search_button)
search_button.click()
And I have have tried all of these with residential proxies, data center proxies and at different timeout lengths, NOTHING works and there is nothing that I can find in the documentation to help with this issue.
does anyone have any insight into this??
I'd understand if this was failing to even sign in but it is failing at the search button, is the page rendered differently for Apify than it is if your running this from your computer maybe?
3 Replies
Someone will reply to you shortly. In the meantime, this might help:
eager-peach•3mo ago
Hey.
In such a context, I would recommend starting with a debug using screenshots when running on the Apify platform and saving them to key-value storage. That way you can get a better understanding of what the problem is. Also test the crawler locally with the same proxy configuration
Any crawler may work differently locally and in the cloud, for example because of proxy (if you didn't use proxy locally). For example Youtube doesn't show some popup windows for me because Ukraine doesn't have GDPR. However, when crawler works with European proxies these popups will.
correct-apricot•2mo ago
Hi, make sure the browser window is displayed in full screen. Also, check whether the XPath you wrote matches the actual structure of the webpage.