Issue with Normalization and Performance in Infinite Query with Large Datasets
I'm using useInfiniteQuery from React Query to handle paginated API data. The data is large (1000+ pages with hundreds of thousands of items), and I normalize it by creating random IDs using crypto.randomUUID(). The issue I’m facing is that every time I fetch a new page, React Query is re-running the normalization for all the data (not just the new page), I am bit worried that it might cause a performance bottleneck as the dataset grows.
Every time a new page is fetched, the select function is running over all previous pages' data, not just the current page, which might make the app slow as the number of pages increases.
What I Expect:
Only the new page's data should be normalized and appended to the existing data.
The previously normalized data should stay untouched in the cache.
The select function should only process the new data, not the whole dataset.
Is it something that can be fixed or am I doing something wrong here.
4 Replies
ratty-blush•9mo ago
The select function should only process the new data, not the whole dataset.select runs over the complete data in the cache, so this is just not how it works. you would have to do this yourself in user-land.
extended-yellowOP•9mo ago
Thanks for your response! Just to clarify, you’re suggesting that I normalize the data directly within the fetch API call, correct?
Also, I have another question: I need to flatten the data returned from each page into a single list. Would it be better to do this inside the select function, or should I handle the flattening within the hook before returning the query data?
ratty-blush•9mo ago
that I normalize the data directly within the fetch API call, correct?yes, with the difference that it will then be stored in that format. but I think that's usually what you want. select had some typing issues before v5, but it works fine for this use-case now
extended-yellowOP•9mo ago
Thankyou very much for the suggestions. I really appreciate it.