inland-turquoise

Continuous re-fetching of data with changes to existing items

Hi There, I have one use-case which I don't really have a good idea how to handle it. The usecase is a photo gallery which updates in "realtime". All photos can (theoretically) change all the time on server side (new photos or changes to existing photos like caption or the status). Any change is indicated by a "lastModified" date for each photo. At the moment without Tanstack Query, I fetch the current list of photos, store it in the state and remember the "highest" last modified date. Every few seconds I fetch the changes since the remembered lastModified date from the server again (it will just return changed ones), go through the results and update the existing items in the state. How to handle this with Query? I don't want to always fetch the complete list again, since normally there are few changes. I looked into "infinite queries" but that looks like more suitable for pages without changes on existing items. I would like to use Query, since I have that everywhere else and I like the other features it has. Any idea or direction? Thx!!

6 Replies

wise-white•2mo ago

It sounds like you just need two endpoints and an optimistic update. 1 endpoint is a list endpoint that would return all your images (or paginate) and another would return individual pictures (or changes like in your case) Your list query is the source of truth in your application, and then you render the updates query with a refetch interval to refetch on your interval you desire: https://tanstack.com/query/latest/docs/framework/react/reference/useQuery#:~:text=to%20a%20string.-,refetchInterval,-%3A%20number%20%7C%20false%20%7C%20((query Then you just need to use either a pull or push approach to seed your infinite query using setQueryData. This blog from Tkdodo does a great job of showing off the two options (push and pull approaches) to implementing queries that have similar data: https://tkdodo.eu/blog/seeding-the-query-cache

Seeding the Query Cache

With suspense for data fetching on the horizon, it is now more important than ever to make sure your cache is seeded properly to avoid fetch waterfalls.

useQuery | TanStack Query React Docs

tsx const { data, dataUpdatedAt, error, errorUpdatedAt, failureCount, failureReason, fetchStatus, isError, isFetched, isFetchedAfterMount, isFetching, isInitialLoading, isLoading, isLoadingError, isPa...

inland-turquoiseOP•2mo ago

Thanks a lot, will look into that! 🙂 Ok, so I understand I would do the following: Have one useQuery with the initial / complete data server URL and queryKey 'list', with a very long stale time(?) to avoid complete re-fetches. Have one useQuery with a query function that uses the lastModified parameter in the server URL and queryKey 'updateQuery'. Then I would use a setInterval / useEffect to 'fetchQuery' the 'updateQuery', merge the data from 'list' with the result and then use setQueryData to update the 'list' query. Is that what you had in mind? 🙂 Is the useEffect / setInterval the right way to do it or should I just define a proper refetchInterval and Query should do it automatically? With staleTime I would do the merging on every render, correct? While via useEffect / fetchQuery I would only do the merging after calling the function. I will try out both and see whats better!

wise-white•2mo ago

I think you can pretty much always just use a refetch Interval instead of a custom interval in a useEffect, since internally this pretty much does exactly the same thing. For the stale-time on your list, yes, you would very likely set this to something pretty high. If you are confident that you won't miss any data from your other query, you could consider making the staleTime infinite and just never refetch unless you want to manually, but often I find setting it to something like 1-2 hours for something like this just guarantees that if there is some bug with the updating logic, you always sync back with the server. With setting the query data, you have a couple options, either a useEffect, or simply setting the query data inside of your queryFn. The benefit of doing it inside the queryFn instead of in an effect is that you never actually require a react re-render to set your data (since you can call the update query without accessing the data, since tanstack query only re-renders arguments that you use). For example, the following code will never cause a react re-render

useQuery({
  queryKey: ["updateQuery"],
  queryFn: async () => {
    const data = await requestUpdates(lastModified);

    const existingData = queryClient.getData(listKey);

    // do something here to update your data

    queryClient.setQueryData(listKey, newData);

    return null;
  },
  refetchInterval: 1000 * 5
})

useQuery({
  queryKey: ["updateQuery"],
  queryFn: async () => {
    const data = await requestUpdates(lastModified);

    const existingData = queryClient.getData(listKey);

    // do something here to update your data

    queryClient.setQueryData(listKey, newData);

    return null;
  },
  refetchInterval: 1000 * 5
})

Personally speaking, if I were building this from scratch, I would rely on a web socket or SSE to receive events from the server when something changes to prevent over fetching and update the list query directly from that socket since I never actually need the update data to be cached in tanstack at all, but obviously this is based on your existing architecture so that may not be an option. Just have a play around with it either way, the main thing you are looking for is setQueryData so that you can just fetch the list once, and then optimistically update the cache whenever you get new data

inland-turquoiseOP•2mo ago

Thanks a lot, really appreciated Websockets is definitely out of my skill league, my backend is PHP on shared wbhosting 😉

wise-white•2mo ago

Haha all good, yeah. Websockets in PHP in particular is difficult because you need a long running process for web sockets, which often means either running some proxy server to keep the connection alive, or running something like franken PHP so that the PHP processes can be long running. Definitely would recommend learning, since web sockets are extremely useful for anything real-time, but yeah, especially in PHP makes sense, and to be honest, the polling approach should be perfectly fine in this case.

inland-turquoiseOP•2mo ago

Yeah, it doesnt have to be really "real time", a delay of a a few seconds is totally ok for my use case 🙂

Gaming

Programming

Continuous re-fetching of data with changes to existing items

Did you find this page helpful?