conscious-sapphire

How to let query be garbage collected but remain in indexddb

And restore the query from indexeddb for it's initialData. I am developing an application that needs a massive amount of data client side

8 Replies

absent-sapphire•5w ago

Look at createPersister

conscious-sapphireOP•5w ago

Ok thanks. All is now working. I have segmented the indexeddb storage per query which has finally fixed the lag in the app. My only issue is that my queryFn looks like this which obviously will not allow new fetches to occur:

export const globalFirmsQueryOptions = () => {
  return queryOptions({
    queryKey: ["globalFirms"],
    queryFn: async ({ queryKey }) => {
      const idbCachedData = await retrieveCachedData({ queryKey })

      if (idbCachedData) {
        return idbCachedData as MinimalFirmData[]
      }

      const responses = await fetchWithAuth(`${env.NEXT_PUBLIC_APP_URL}/api/firms`)

      const firms = await responses.json()

      return firms as MinimalFirmData[]
    },
    refetchInterval: 60 * 1000,
    staleTime: 60 * 1000,
    gcTime: 0,
    refetchOnMount: false,
    refetchOnWindowFocus: false,
  })
}

export const globalFirmsQueryOptions = () => {
  return queryOptions({
    queryKey: ["globalFirms"],
    queryFn: async ({ queryKey }) => {
      const idbCachedData = await retrieveCachedData({ queryKey })

      if (idbCachedData) {
        return idbCachedData as MinimalFirmData[]
      }

      const responses = await fetchWithAuth(`${env.NEXT_PUBLIC_APP_URL}/api/firms`)

      const firms = await responses.json()

      return firms as MinimalFirmData[]
    },
    refetchInterval: 60 * 1000,
    staleTime: 60 * 1000,
    gcTime: 0,
    refetchOnMount: false,
    refetchOnWindowFocus: false,
  })
}

I have not taken advantage of createPersister however - it's interesting that you can enable this per query. I enabled this by adjusting the behavior of PersistQueryClientProvider in my custom implementation I think with createPersister it would be the same kind of issue where if the query gets GC'ed then it would be removed from the storage correct? I want to be able to restore from idb if the data exists in there

absent-sapphire•5w ago

No, createPersister decouples GC from external storage. You can have a query on disc but not in memory, that's the biggest difference

conscious-sapphireOP•5w ago

You're the man. Last question on the topic that's posing an issue with this refactor if you have the availablility: I have a query that is mounted only while the user is in "edit" mode. When the user is finished editing I trigger a mutation. In the onSettled of the mutation I attempted to call each of fetchQuery, invalidateQuery, refetchQuery (individually) for the query in question that is only mounted while the user is in 'edit' mode. I have found that none of these functions on their own trigger a refetch of the data. I must call fetchQuery, then invalidateQuery, then fetchQuery once again to get the updated data. Does this align with your expectations? The query that only is mounted while the user is in edit mode has gcTime = 0 (for good reason) and is persisted with the createPersister helper Have just found that calling fetchQuery twice does the trick also (with staleTime: 0)

absent-sapphire•5w ago

Does this align with your expectations?

no 😂

absent-sapphire•5w ago

if the querry isn't active, invalidate won't refetch it. But fetchQuery and refetchQuery will always refetch it IF it's stale. If it's not in memory we'll go to the disk and trigger a re-fetch after that. You can try setting refetchOnReconnect: 'always' but if the query is stale it should happen automatically: https://github.com/TanStack/query/blob/846d53d98992d50606c40634efa43dea9965b787/packages/query-persist-client-core/src/createPersister.ts#L215-L220

GitHub

query/packages/query-persist-client-core/src/createPersister.ts at ...

🤖 Powerful asynchronous state management, server-state utilities and data fetching for the web. TS/JS, React Query, Solid Query, Svelte Query and Vue Query. - TanStack/query

conscious-sapphireOP•4w ago

Haha alright well believe it or not the following code does not trigger any network request. Simply adding another fetchQuery above the first actually will in my scenario. The staleTime on the queryOptions is always 0, nowhere in the code is it set to be anything other than 0.

    case "investments": {
      const updated = await queryClient.fetchQuery({
        ...investmentsQueryOptions({ firmId }),
        staleTime: 0,
      })

      await syncResourceToGlobalCache(
        queryClient,
        firmId,
        updated,
        investmentsQueryOptions({ firmId: undefined }).queryKey
      )
      break
    }

    case "investments": {
      const updated = await queryClient.fetchQuery({
        ...investmentsQueryOptions({ firmId }),
        staleTime: 0,
      })

      await syncResourceToGlobalCache(
        queryClient,
        firmId,
        updated,
        investmentsQueryOptions({ firmId: undefined }).queryKey
      )
      break
    }

In the above case updated will just be the old data and no fetch triggered. In the following case updated is the new data and fetch is triggered. Of course this isn't a reproduction so don't expect any follow up here. But pretty weird stuff

    case "investments": {
      await queryClient.fetchQuery({
        ...investmentsQueryOptions({ firmId }),
        staleTime: 0,
      })

      const updated = await queryClient.fetchQuery({
        ...investmentsQueryOptions({ firmId }),
        staleTime: 0,
      })

      await syncResourceToGlobalCache(
        queryClient,
        firmId,
        updated,
        investmentsQueryOptions({ firmId: undefined }).queryKey
      )
      break
    }

    case "investments": {
      await queryClient.fetchQuery({
        ...investmentsQueryOptions({ firmId }),
        staleTime: 0,
      })

      const updated = await queryClient.fetchQuery({
        ...investmentsQueryOptions({ firmId }),
        staleTime: 0,
      })

      await syncResourceToGlobalCache(
        queryClient,
        firmId,
        updated,
        investmentsQueryOptions({ firmId: undefined }).queryKey
      )
      break
    }

I think the first fetchQuery is what restores the data from disk and then the second is what triggers the refetch. I have played around with the internal source code and been unable to have the ability to properly restore from disk, when switching tabs for ex, and simultaneously have the ability to use fetchQuery on a query that exists on disk but not in the queryCache to get fresh data by triggering a network request removing the persister in this fetchQuery invocation also works... but unsure what kind of impact that will have. I don't want to disable persistence aside from in this special case

await queryClient.fetchQuery({
    ...investmentsQueryOptions({ firmId }),
    persister: undefined,
    staleTime: 0,
})

await queryClient.fetchQuery({
    ...investmentsQueryOptions({ firmId }),
    persister: undefined,
    staleTime: 0,
})

this seems to be where I'm landing:

      await idbQueryPersister.restoreQueries(queryClient, {
        queryKey: fundsQueryOptions({ firmId }).queryKey,
      })
      const updated = await queryClient.fetchQuery(fundsQueryOptions({ firmId }))

      await idbQueryPersister.restoreQueries(queryClient, {
        queryKey: fundsQueryOptions({ firmId }).queryKey,
      })
      const updated = await queryClient.fetchQuery(fundsQueryOptions({ firmId }))

absent-sapphire•4w ago

I can take a look if you out that into a simple, standalone reproduction (preferably stackblitz)

Gaming

Programming

How to let query be garbage collected but remain in indexddb

Did you find this page helpful?