Help with website-content-crawler for specific website
Hi all, I'm an ML engineer but complete beginner using Apify. It looks great in theory but for my first project I ran into the roadblock that I cannot crawl this website properly: https://georgian.io/resources/
There are a bunch of articles and blogs there for which I want the text but running the web-crawler only returns the main page without any sub-links, regardless of the max-depth I specify.
I essentially followed the langchain tutorial here: https://apify.com/apify/website-content-crawler.
Simply replacing the url. I then also manually set the max depth in the website for the same call and still got a single page as output.
Any advise? Thanks!!
Georgian
Resources
Find resources from Georgian by topic (applied AI, trust, blockchain, conversational AI and more) or by type of content (video, podcast, white papers...)
1 Reply
ratty-blush•3y ago
Hey)
Can you please provide URL to your run on the platform, so I can check logs/your input to understand what is happening?