HTML code via dev google tools and soup are different
Hello,
I'm currently working on a small project which aims to scrap articles on the subject of crypto (or other is an example)
I've already done some scraping in the past but on simple sites.
You can refer to my result, why the articles are not loaded in the soup ? (let's focus on the script coinmarketcap even if I have the same problem with the news tab of google ...)
expected result: https://coinmarketcap.com/community/articles/browse/?sort=algo&page=1&category=
result obtained:
Cryptocurrency updates from projects across the web
See the latest cryptocurrency updates from projects in all sectors of crypto, such as DeFi, Play to Earn, GameFi, NFTs and more.
1 Reply
Hi.
You are not getting articles because this data is not in the html code at the link - https://coinmarketcap.com/community/articles/browse/?sort=algo&page=1&category=
If you pay attention, the site is built on JS and the data is loaded dynamically. The link to get the content is https://api.coinmarketcap.com/content/v3/news/cryptown?page=1&size=100&sort=algo