Hey all, with the introduction to the
Hey all, with the introduction to the new
/map endpoint, we have the need to list the URLs returned and then have the user click to select the the URLs they want crawled/scraped.
Sending one /scrape endpoint per URL seems inefficient on both sides and will blow through our rate limit.
It seems like includedPaths does not only scrape the included paths as a part of the /crawl endpoint.
What is the recommended way to batch scrape a list of URLs?9 Replies
We are working on a batch endpoint to improve this! 🙂
Thanks @Adobe.Flash! Do you know ETA on this feature?
We are actively working on the feature above and it'd be good to know so we can prioritize
Np! and not yet, but we are aiming for around next week (the earliest)!
Appreciate the response, let me know if there is anywhere I can track development or if it will be sent out as a product email!
We'll probably just hold off on shipping the feature until this is live
Hey @Adobe.Flash any update on the batch endpoint?
Hey @Cole! Yes! Coming next week!
Appreciate the quick reply, thank you!
@Adobe.Flash Checking back in here for ETA?
Hey @Cole , we are still finishing its load testing / scalability tests this week. We want to make sure we can support it without any impact. If youd like I can provide you with beta access to it on Wednesday!
Hey @Cole, quick update, we might have something for you tomorrow afternoon. Will keep you updated
Thanks @Adobe.Flash, let me know!
Hey @Cole check your DM! Just sent you info about the batch scrape!