hi guys
Hello there,
I'm reaching out for assistance with data scraping. Once completed, is there a method to structure the data according to my requirements? Additionally, is there a way to eliminate replicated data and retain only one instance? Your expertise in this matter would be greatly appreciated. Thank you.
1 Reply
national-gold•14mo ago
Hi, on Apify platform, you can store your data either to dataset or key value store. You can store any valid JSON in a dataset, so you can structure your data any way you want. Does this answer you question?
Regarding elimination of duplicates, we usually keep track of IDs (e.g.
HashSet) of items that we have already pushed to dataset. Everytime we're pushing a new item, we check whether it's already been pushed.Dataset | Platform | Apify Documentation
Store and export web scraping, crawling or data processing job results. Learn how to access and manage datasets in Apify Console or via API.