API Performance
🍼Noob
Typescript
NextJS
Tailwind/CSS
TypescriptNextJSTailwind/CSSI made an API to get some data from CSVs into my web app to be searched through and used. The issue I have been trying to tackle is that there are thousands of items, many CSVs files, and the request (yes, just one) was too large and the page would take a long time to load (25s on fast 4G, 2.4min on slow 4G, and 8min on 3G). At first I didn't think much, "that's fair, I am doing quite a lot and this is naive, let me try optimizing it a bit". I tried pagination and sending the data in chunks and also streaming the data as it's being read. It was even slower than before with this. That didn't make any sense to me. I figured that maybe the CSV parsing is too slow so I just parsed it and turned the data into json directly and have been using that and am streaming the data as it is being read but even that feels slow. I'm not really sure what I am doing wrong and I would like some direction that might help the performance of the API.
