How to handle a huge Json file?
I have huge json file with many objects from Apify.com, each object is a business, the file is too big and when I request it in the browser in order to save the results to mysql it's waiting for hours to load the file. What should I do? I need to process the file before insert it to mysql.
4 Replies
eastern-cyan•3y ago
Are you downloading it or opening in new tab? Downloading expected to work even for huge files, if its too big to process consider to split by https://apify.com/lukaskrivka/dedup-datasets and process data by chunks
subsequent-cyanOP•3y ago
I open it in a new tab, when I try to open it in vs code, vs code crashed each time. I will check this. thanks!
eastern-cyan•3y ago
New tab will be browser killer for sure, you need to dll file and then use notepad or sublime to open it, basically anything wo formatting or linting applied on opening, not vscode exactly because of post-processing, it works fine only for regular file sizes.
other-emerald•3y ago
You should either download it in chunks or post-process it in actor with the actor Alexey suggested. If it is just one file, load it inside of actor and split it into many files or put it to dataset that you can paginate