Crawl Is Scraping Website Too
I've had a service running for a while, that uses the /crawl endpoint to crawl 15 pages of a website and has run with no issue for some time.
Recently I've noticed that it is also scraping each of the 15 pages individually, as well as doing a separate crawl. This is causing us to use double the number of credits per website.
Is this a know issue or knock-on from a recent change?
Any help would be appreciated.
Recently I've noticed that it is also scraping each of the 15 pages individually, as well as doing a separate crawl. This is causing us to use double the number of credits per website.
Is this a know issue or knock-on from a recent change?
Any help would be appreciated.
