Syncing Paginated Rest API
Is there a recommended pattern for syncing a paginated rest API? The example in the docs for an API assumes all items are returned in a single request.
I have some ideas, and I know the answer is highly depended on how your API works, but any examples would be helpful to make sure I'm going about it in roughly the right way.
7 Replies
metropolitan-bronze•3mo ago
this is on our TODO to investigate — our hope is basically to copy the tanstack/query approach but we need to investigate exactly how that'd work especially tying it into the query interface
what are your ideas?
robust-apricotOP•3mo ago
The quick and easy way would be to loop through the endpoint in the queryfn. My API is cursor based, so just a while loop until cursor is null, handling retries etc.
A nicer way would be to have some kind of adapter that knows how to fetch a next page, comparable to the useInfiniteQuery hook in tanstack query.
This is a semi-related point, I was reading through the issue on offline support, and I believe it was you who mentioned that the cache will likely live in the syncing layer, not in Tanstack DB. The only issue I see with that is if you don't want to resync all the data. Maybe you just want to fetch items filtered by "lastChanged" or something like that, and that would live in the queryFn or somewhere else in Tanstack DB
metropolitan-bronze•3mo ago
ah interesting — like you'd want to eagerly paginate through your API?
The only issue I see with that is if you don't want to resync all the data. Maybe you just want to fetch items filtered by "lastChanged" or something like thatThat kinda illustrates my point 😆 how would DB know about this logic? If you wanted to re-fetch based on lastChanged, you'd probably want a custom collection with your specific syncing logic embedded it's similar to how Query lets you do custom stuff like this but you have to bail out of the easy path and manually manipulate the cache. In a similar way, if you want custom syncing logic, you create a custom collection (which could still be based on the query collection)
robust-apricotOP•3mo ago
Yeah, eagerly paginate through it is correct.
I see your point about caching, I guess there just isn't an easy solution especially with the amount of variance around apis
metropolitan-bronze•3mo ago
Yeah, that's half the point of a sync engine like electric, provide some strong efficient patterns for common ways of doing things
robust-apricotOP•3mo ago
Yeah, I'm realizing I'm probably going to end up rebuilding electric if I go down this path haha
So I'm going to reconsider
metropolitan-bronze•3mo ago
One of our goals with DB is you're not forced into any particular paradigm. If the data is easy to sync then sync by all means. If not, then just fetch