Elements not rendering
Hi, I am trying to crawl a page (it's a very small crawl, I used to do it manually every week).
I am using the PlaywrightCrawler.
I am running into a problem, when visiting the site I want to crawl, the elements that are most important to crawl are not appearing.
They are not hidden with CSS or something, they are just literally not in the DOM. The website uses server side rendering.
I thought that it may have to do something with screen size. So I decided to disable headless, increase browser and page viewport to 1920x1080 and try it again. However, the elements were still not being shown.
I tried adjusting my viewport in a different browser (not crawling, just browsing), and it shows the data on every viewport (even the tiniest I can make it).
Next I tried visiting the site on my phone, and the data was not showing. Could it be that it detects the crawler as a phone or something? I didn't turn anything regarding this on.
6 Replies
frail-apricotOP•2y ago
By the way I had this problem using a python crawler too but I figured it may have been the package, that's why I hopped to this.
Some other info that may be quite handy, the elements that are not appearing have an ID, and there is exactly one other element before it with the exact same ID. (So, every second element, the ones that I need, with the same ID for the first time, are not appearing).
But since this is not a problem for regular browsers I thought it would be fine for this too? I know that you arent supposed to have duplicate ids (it aint my site tho)
I am also noticing now, that if I go to the page in a private window on a regular browser, the data is not being shown either.
@𝙉𝙞𝙘𝙠 just advanced to level 1! Thanks for your contributions! 🎉
frail-apricotOP•2y ago
I think it has to do something with the cookies, I am going to look how I can change the cookies to something i want myself
Yup. Loaded the site on another device and it looks like the cookies or localstorage get sent to the server and based on the values (like last visited) it hides some elements. So that's it..
frail-apricotOP•2y ago
Getting this error now when setting localstorage
frail-apricotOP•2y ago
frail-apricotOP•2y ago
Oh wait that's because its not loaded on an url yet haha
yes now its working :chikajam: