Change Tracking Cache TTL Rate Limiting Question (Free Tier)
Hey Firecrawl team! 👋
I'm building a system that needs to periodically check websites for changes (e.g., every 10 minutes) and then notify users if something is different. Your Change Tracking feature seems perfect for this!
To test how well this periodic checking would work, I tried it out on the Free tier. I set up a script to poll
https://api.github.com/zen every 5 minutes using fetch(..., mode="GitDiff"), since that URL gives fresh content every time.
My script ran, but the problem I hit was that Firecrawl kept reporting changeStatus: "same" for about 17 hours straight, even though the actual Zen quote was changing constantly. It seems Firecrawl was serving a cached version for that whole time. Only after ~17 hours did it finally do a fresh scrape and report a change.
This long cache delay makes it really difficult to test if my periodic checking logic is working correctly during development, as I can't simulate frequent changes being detected.
Is this ~17-hour cache window (TTL) the expected behavior for the Free tier? Is there any way to get a faster refresh or bypass the cache just for development/testing purposes, so I can verify my system handles detected changes properly without waiting so long?
Thanks for any clarification!3 Replies
more details on the issue here
this seems to be a library bug?
Hey Alex! Thanks, we're looking at the library issue. Firecrawl does not use caching, will look into this.