worker to allow web scrape by friendly bot?
I'm using an AI app called Mendable to scrape my own website for training. CF is blocking their scraping bot when it tries to procedurally visit every page based on the sitemap. Mendable claims it would be "impossible" to know the IP range for their scraper, so I don't think I can whitelist the bot.
Is there a way to write a simple worker that would allow this "friendly bot" to scrape my website?
Is there a way to write a simple worker that would allow this "friendly bot" to scrape my website?
