Hi, I am building a webscraper and I
Hi, I am building a webscraper and I want to use a kind of persistent cache to determine which links I have scraped recently in other run-throughs and which not. Is the best solution for this to use the KeyValueStore .getValue and KeyValueStore .setValue during the requestHandlers ?
1 Reply
conscious-sapphire•13mo ago
best solution is named RequestQueue, KVStore might be better if you have advanced logic, i.e. if you want more stats aside just checkup if request already processed