Crawlee not working(?) on a page with shadow dom
Hey, I've encountered a website using shadow dom, where crawlee isn't able to find elements (for a good reason).
https://developer.mozilla.org/en-US/docs/Web/Web_Components/Using_shadow_DOM
I was wondering since there is no mentions of shadow dom if anyone knows what to look at to make it work?
8 Replies
flat-fuchsiaOP•3y ago
It looks like this might be the answer https://stackoverflow.com/questions/64368789/puppeteer-clicking-button-in-shadowroot, going to try this with crawlee
Stack Overflow
puppeteer: clicking button in shadowroot
I'm having difficulties with actions on elements in shadowroot in my test context. Lets say I have a web component <my-component /> and it contains a button <input id="my-button"...
flat-fuchsiaOP•3y ago
Okay, that doesn't work. Chromiu has the "Copy JS path" option disabled for that element.
flat-fuchsiaOP•3y ago
official Puppeteer discussion https://github.com/puppeteer/puppeteer/issues/858
GitHub
Query nodes within shadow roots · Issue #858 · puppeteer/puppeteer
Currently, we don't have a way to find/query nodes with shadow roots. Users need to traverse .shadowRoots themselves. It would be ideal if the DTP supported an option to make this easier, b...
flat-fuchsiaOP•3y ago
Looks like as long as the shadow root is attached with the
closed option, it will be inaccessible from the page's JS, neat.@Michal just advanced to level 1! Thanks for your contributions! 🎉
flat-fuchsiaOP•3y ago
Since the page is opening a consent dialog in the shadow dom, I can just ignore it for now. But if the whole page was encased int it that might be a problem.
flat-fuchsiaOP•3y ago
One way to get through to the shadow dom would be to use Chrome DevTools https://chromedevtools.github.io/devtools-protocol/tot/DOM/
Chrome DevTools Protocol
Chrome DevTools Protocol - version tot - DOM domain
flat-fuchsiaOP•3y ago
Someone even used the DevTools protocol to intercept traffic, very cool https://jsoverson.medium.com/using-chrome-devtools-protocol-with-puppeteer-737a1300bac0