Creating multiple collections from a single query?
Hiya folks! I'm curious: we're looking at adding batched rpc routes for a given page that return all data required to render said page in a single request. Is there a way to create/update multiple collections from a single query?
8 Replies
continuing-cyan•7d ago
two ways — one is have a mass collection & then query it to split it out into per-type collections
continuing-cyan•7d ago
or you can use the Direct Writes API https://tanstack.com/db/latest/docs/collections/query-collection#direct-writes
Query Collection | TanStack DB Docs
Query Collection Query collections provide seamless integration between TanStack DB and TanStack Query, enabling automatic synchronization between your local database and remote data sources. Overview...
ratty-blushOP•6d ago
gotcha, thank you! i think the direct writes API is what we’ll reach for.
out of curiosity, on the underlying differential dataflow engine: is it demand-driven/lazy recompute or is it eager?
(for context, i work on rust-analyzer and spent a lot of time with its underlying incremental computation library, salsa, which is lazy/demand-driven.)
continuing-cyan•6d ago
@samwillis is the one to nerd out with you on this 😆 but collections themselves are lazy (only run w/ there's an active query) but updates are eagerly/sync processed as e.g. you enter something in a form you need the UI to re-render immediately.
ratty-blushOP•6d ago
on that’s an interesting split! yeah, salsa is lazy (you only really need updates wherever your cursor is!) but then lsp grew features like “pull diagnostics” and “semantic highlighting” where it kinda makes sense to do things eagerly. would love to learn from sam as to how y’all split the difference 🙂
extended-salmon•6d ago
I wrote a thread here we a few details of some of the optimisations we make: https://x.com/samwillis/status/1970141020563337663
but yes, they lazily start, minimise the data processed on start, the eagerly process data as their sources change.
We're close to adding lazy pagination to them. So a query with an orderBy and limit is already optimised, but we will make the offset+limit mutable, so you can page through and it will pull in data as needed. Perfect for virtualised scrolling where you have a window over the results.
Sam Willis (@samwillis)
Live queries in @tan_stack DB are incremental - they are maintained, rather than re-run, when data changes.
It does this with differential dataflow / DBSP.
Usually this tech trades fast updates for a slower start.
🧵 on how we made *both* initial and update runs lightning fast!
X
wise-white•5d ago
Lazy pagination sounds amazing! Will it be available with Electric collections? I have trouble understanding how this might be possible.
extended-salmon•5d ago
Yep, going to be supported by both query and electric (and any others that implement it)
Sync from electric will become incremental to support it - initially sync what's needed for the start of the query, then progressively sync more in the background.