CA
Crawlee & Apify•2y ago
harsh-harlequin

Pupeteer unable to find element (dev tools show the element)

Hello everyone, I'm currently working on a web scraping tool that extracts data from the following web page: https://www.handelsregister.de/rp_web/documents-dk.xhtml. However, I'm running into some unexpected issues and I'm not sure how to debug them. Here's a brief overview of the issue: - I'm using Puppeteer to scrape data from the web page. - The tool is able to navigate to the correct page and locate the element that I want to click. - However, when I try to extract the element using document.querySelectorAll(), I'm not getting the expected results. - Specifically, I'm using the following code to extract the first toggle span: 
const firstToggleSpan = await page.evaluate(() => {
  const toggleSpans = document.querySelectorAll(
    "span.ui-tree-toggler.ui-icon.ui-icon-triangle-1-s",
  );
  return toggleSpans[0]; // prints `{}` to   the server logs
});
const firstToggleSpan = await page.evaluate(() => {
  const toggleSpans = document.querySelectorAll(
    "span.ui-tree-toggler.ui-icon.ui-icon-triangle-1-s",
  );
  return toggleSpans[0]; // prints `{}` to   the server logs
});
- However, firstToggleSpan is an empty object, - whereas running the same code in the Puppeteer DevTools console, it returns the element that I want to click. How can I debug pupeteer's page selectors and why doesn't it behave as the querySelector in the dev tools?
No description
2 Replies
harsh-harlequin
harsh-harlequinOP•2y ago
for whatever reason the handler functions didn't work. However, page.evaluate() did the trick in the end. 
  const firstToggleSpan = await page.evaluate(() => {
    const toggleSpans = document.querySelectorAll<HTMLSpanElement>(
      "span.ui-tree-toggler.ui-icon.ui-icon-triangle-1-e",
    );

    if (!toggleSpans.length) {
      throw new Error(
        "Could not locate the toggle for `Documents on register number`",
      );
    }
    // toggle
    toggleSpans[0]?.click();
  });
  const firstToggleSpan = await page.evaluate(() => {
    const toggleSpans = document.querySelectorAll<HTMLSpanElement>(
      "span.ui-tree-toggler.ui-icon.ui-icon-triangle-1-e",
    );

    if (!toggleSpans.length) {
      throw new Error(
        "Could not locate the toggle for `Documents on register number`",
      );
    }
    // toggle
    toggleSpans[0]?.click();
  });
abundant-blush
abundant-blush•2y ago
Hey there, the problem is you're trying to pass element from pippeteer context into node context, which is not possible afaik (if I remember correctly - because it's not stringifiable). So yeah - you should either return element using built-in puppeter method (https://pptr.dev/api/puppeteer.page._). Or otherwise as you mentioned in the next message - you could manipualte the element inside of the page.evaluate()

Did you find this page helpful?