CA
Crawlee & Apify•3y ago
genetic-orange

SyntaxError Unexpected end of JSON input!

Hi! Does anyone know what do I get this error and How can I solve it? I noticed it happens when I have many files loaded in the directory. Is there a way to get past this? Hereis the full log message: 
 SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at findOrCacheDatasetByPossibleId (C:\Users\misag\OneDrive\Documents\Joiakim\Neontech\my-crawler\node_modules\@crawlee\memory-storage\cache-helpers.js:48:39)
    at async DatasetClient.get (C:\Users\misag\OneDrive\Documents\Joiakim\Neontech\my-crawler\node_modules\@crawlee\memory-storage\resource-clients\dataset.js:79:23)
    at async StorageManager._getOrCreateStorage (C:\Users\misag\OneDrive\Documents\Joiakim\Neontech\my-crawler\node_modules\@crawlee\core\storages\storage_manager.js:97:33)
    at async StorageManager.openStorage (C:\Users\misag\OneDrive\Documents\Joiakim\Neontech\my-crawler\node_modules\@crawlee\core\storages\storage_manager.js:73:35)
    at async addPrestashop (file:///C:/Users/misag/OneDrive/Documents/Joiakim/Neontech/my-crawler/src/main.js:51:18)
 SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at findOrCacheDatasetByPossibleId (C:\Users\misag\OneDrive\Documents\Joiakim\Neontech\my-crawler\node_modules\@crawlee\memory-storage\cache-helpers.js:48:39)
    at async DatasetClient.get (C:\Users\misag\OneDrive\Documents\Joiakim\Neontech\my-crawler\node_modules\@crawlee\memory-storage\resource-clients\dataset.js:79:23)
    at async StorageManager._getOrCreateStorage (C:\Users\misag\OneDrive\Documents\Joiakim\Neontech\my-crawler\node_modules\@crawlee\core\storages\storage_manager.js:97:33)
    at async StorageManager.openStorage (C:\Users\misag\OneDrive\Documents\Joiakim\Neontech\my-crawler\node_modules\@crawlee\core\storages\storage_manager.js:73:35)
    at async addPrestashop (file:///C:/Users/misag/OneDrive/Documents/Joiakim/Neontech/my-crawler/src/main.js:51:18)
2 Replies
genetic-orange
genetic-orangeOP•3y ago
I found the issue; when using openDataset() on empty json file or when json.lock file are still not resolved in Dataset. I don't know if I said this right; but to resolve this error you have to delete the empty data files and remove json.lock files.
conscious-sapphire
conscious-sapphire•3y ago
@Joaquim How did you generate this corrupted state? If it was by normal previous crawling, that sounds like a bug.

Did you find this page helpful?