Change Tracking
As I understand it, the current change tracking scrapes the url content and then reports whether its changed or not. Is there a way to check for a change or when the site was last updated without fully scraping the url content again?
3 Replies
I'm curious why you're trying to avoid doing a full scrape. Are you concerned about the credit cost, or something else?
Realistically, the only way to know if something has changed is to fully scrape it (which is cheap), and then compare it against a previous scrape.
i'm mainly concerned about speed / latency in my application. firecrawl is consistenly taking 10-20 seconds per url. it'd be ideal if i could get it down to a few seconds by maybe storing the data after scraping it and then just having some tool that can quickly check whether i need to rescrape or just return what's in the database
Do you have a fixed set of URLs that you are scraping? If so, you could consider scraping them on a schedule, and then always return the result in your DB when you are asked for it.
I'd need to know a lot more about your use case and requirements to be more helpful.