❔ [Unity-ish/Newtonsoft] Efficient ways to deserialize a large amount of json data.
Update: This problem appears to be a non-issue in builds.
I am going to figure out what the heck the editor is doing here vs builds. But not right now because work.
Problem: It seems that anytime I attempt to deserialize data using the Newtonsoft tools, I'll incur a pretty nasty spike on the CPU and hitch even in an empty project. The API/Endpoints I'm stuck with do not have any data-chunking afaik, and also routinely return data in one large array of objects. In production, I'll be expected to handle up to ~15000 objects from an endpoint.
Code can be viewed here https://pastebin.com/h6qycPmv
Json can be pulled directly from this endpoint: https://jsonplaceholder.typicode.com/comments
Methods 1 :
Coroutine (Picture Labeled A)
Unity Coroutine, reaching out to grab the data, then deserializing it to a defined class:
Note: With this, I was noticing a pretty gnarly spike. Figured I wasn't doing something correctly or was simply abusing deserialization and not using it correctly. I started researching other ways of how to consume JSON data from an endpoint.
Methods 2:
Two variations (pictures B and C). Picture B is the same JSON as Picture A, while Picture C is a trimmed down version.
Goal here was to deserialize the json one element at a time using a Filestream. I had a hunch to add a manual delay in there to add a gap between each deserialization.
What's most curious to me is that even with this, I'm incurring that same general cpu hit. Which leads me to believe I'm implementing this stuff totally incorrectly. <-- Likely as hell.
Is there not some relatively efficient way to deserialize a mass of incoming objects? Or at least process json from an endpoint without hitching?
Edit: also aware of some bad practices here to include the async void stuff, but trying to triage
I am going to figure out what the heck the editor is doing here vs builds. But not right now because work.
Problem: It seems that anytime I attempt to deserialize data using the Newtonsoft tools, I'll incur a pretty nasty spike on the CPU and hitch even in an empty project. The API/Endpoints I'm stuck with do not have any data-chunking afaik, and also routinely return data in one large array of objects. In production, I'll be expected to handle up to ~15000 objects from an endpoint.
Code can be viewed here https://pastebin.com/h6qycPmv
Json can be pulled directly from this endpoint: https://jsonplaceholder.typicode.com/comments
Methods 1 :
Coroutine (Picture Labeled A)
Unity Coroutine, reaching out to grab the data, then deserializing it to a defined class:
Note: With this, I was noticing a pretty gnarly spike. Figured I wasn't doing something correctly or was simply abusing deserialization and not using it correctly. I started researching other ways of how to consume JSON data from an endpoint.
Methods 2:
Two variations (pictures B and C). Picture B is the same JSON as Picture A, while Picture C is a trimmed down version.
Goal here was to deserialize the json one element at a time using a Filestream. I had a hunch to add a manual delay in there to add a gap between each deserialization.
What's most curious to me is that even with this, I'm incurring that same general cpu hit. Which leads me to believe I'm implementing this stuff totally incorrectly. <-- Likely as hell.
Is there not some relatively efficient way to deserialize a mass of incoming objects? Or at least process json from an endpoint without hitching?
Edit: also aware of some bad practices here to include the async void stuff, but trying to triage
Pastebin
Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
