xenial-black

typing in helpers in db-ivm

Hey! I'm using db-ivm to aggregate events from an event-sourced system for display. Working great so far! I've noticed a lot of boilerplate around sorting and filtering, though, and so I've tried to make myself a helper:

export function build<Event extends { timestamp: number }, Out, Key extends string | number>(
  builder: (events: Event[]) => Out | null | undefined,
) {
  return reduce<Key, Event, Out, Keyed<Key, Event>>((events: [Event, number][]) => {
    const res = builder(
      events
        .filter(([_, mult]) => mult === 1)
        .map(([ev, _]) => ev)
        .toSorted((a, b) => a.timestamp - b.timestamp),
    )

    if (res) {
      return [[res, 1]]
    } else {
      return []
    }
  })
}

export function build<Event extends { timestamp: number }, Out, Key extends string | number>(
  builder: (events: Event[]) => Out | null | undefined,
) {
  return reduce<Key, Event, Out, Keyed<Key, Event>>((events: [Event, number][]) => {
    const res = builder(
      events
        .filter(([_, mult]) => mult === 1)
        .map(([ev, _]) => ev)
        .toSorted((a, b) => a.timestamp - b.timestamp),
    )

    if (res) {
      return [[res, 1]]
    } else {
      return []
    }
  })
}

This works fine, except that I have to manually specify the type parameters or they're inferred to be never. For example, the type params in this call are required:

build<
  QuestionTitleSetV1 | QuestionAddedV1,
  { title: string; type: QuestionAddedV1['data']['type'] },
  string
>((events) => {
  // …
})

build<
  QuestionTitleSetV1 | QuestionAddedV1,
  { title: string; type: QuestionAddedV1['data']['type'] },
  string
>((events) => {
  // …
})

I see an addOperator call to graph—is that a better way to add helpers like this? Otherwise, any tips on getting type inference working correctly?

28 Replies

wise-white•3d ago

curious why not just use DB directly? db-ivm is very flexible so hard to tame down the complexity which is why DB exists (in part) so it can generate really precise/fast ivm code

xenial-blackOP•3d ago

Can it make aggregates out of events? I didn’t see anything like that in the docs. It seemed pretty record-oriented

wise-white•3d ago

https://tanstack.com/db/latest/docs/guides/live-queries#groupby-and-aggregations

Live Queries | TanStack DB Docs

TanStack DB Live Queries TanStack DB provides a powerful, type-safe query system that allows you to fetch, filter, transform, and aggregate data from collections using a SQL-like fluent API. All queri...

wise-white•3d ago

let us know if something is missing!

xenial-blackOP•3d ago

Sounds like I need to read up! Thanks for the pointer So overall it seems like this package assumes you want writes to sync immediately. I’m designing this for the write authority to be on the client, though. The events are stored in IndexedDB and really only stored in Electric for browser resets and device syncing. Is there some pattern that you think would work well in db to handle this?

wise-white•3d ago

you can "persist" the data to indexedb like https://tanstack.com/db/latest/docs/overview#localstoragecollection actually would love PRing an indexeddb version of that then store txs as the events to sync around

xenial-blackOP•3d ago

would you mind expanding on the last part, actually? do you mean storing sync state alongside the events? I guess then I’d make manual transactions based on that data?

wise-white•3d ago

Yeah, create a custom transaction type that stores the txs when committed

xenial-blackOP•11h ago

hmm, I'll look at that. Thanks. ok, so I tried this out @Kyle Mathews and… is there some aggregation you had in mind here? I don't see a way to reduce a group in the db API 🤔

export const pingEventCollection = createLiveQueryCollection({
  id: 'ping-events',
  query: (q) =>
    q
      .from({ event: eventCollection })
      .where(({ event }) => eq(event.type, 'PingAdded'))
      .groupBy(({ event }) => event.data.time)
      // what goes here to reduce?
})

export const pingEventCollection = createLiveQueryCollection({
  id: 'ping-events',
  query: (q) =>
    q
      .from({ event: eventCollection })
      .where(({ event }) => eq(event.type, 'PingAdded'))
      .groupBy(({ event }) => event.data.time)
      // what goes here to reduce?
})

It looks like .fn.select only works on a single row at a time (sure) and there's no custom aggregation… is there?

wise-white•11h ago

You see the aggregation functions?

wise-white•11h ago

https://tanstack.com/db/latest/docs/guides/live-queries#aggregate-functions

Live Queries | TanStack DB Docs

xenial-blackOP•11h ago

yes, I read that. But I don't think those fit what I'm after. I'm working with event data here, and the aggregates are actually a custom reduction. Is there an aggregate function where I can run some function that just gets a group and returns whatever? lemme give an example. I have these two events (simplified):

type AddQuestion = { type: "AddQuestion", data: { id: string, type: "tag" | "number" } }
type SetQuestionName = { type: "SetQuestionName", data: { id: string, name: string } }

type AddQuestion = { type: "AddQuestion", data: { id: string, type: "tag" | "number" } }
type SetQuestionName = { type: "SetQuestionName", data: { id: string, name: string } }

and I want to produce an aggregate that looks like this:

type Question = {
  type: "tag" | "number"
  name: string
}

type Question = {
  type: "tag" | "number"
  name: string
}

I can where and then groupBy (subject to ignoring an error about type narrowing) but in order to produce the desired output type I need something like db-ivm's reduce. that's why I was using that instead of db directly in the first place. 😅

xenial-blackOP•11h ago

basically if I could make a custom one of these I think I'd be in business. https://github.com/TanStack/db/blob/c4c2399cbe0969da01714ef59f4aa05ffdcb75dd/packages/db-ivm/src/operators/groupBy.ts#L147-L163

GitHub

db/packages/db-ivm/src/operators/groupBy.ts at c4c2399cbe0969da0171...

A reactive client store for building super fast apps - TanStack/db

wise-white•11h ago

cc @samwillis seems like a fn.reduce function would be useful

xenial-blackOP•11h ago

if a hypothetical fn.reduce got a signature like (in: T[]) => U[] I think it might also be an escape hatch for type narrowing like we were talking about in the other thread.

deep-jade•11h ago

Ooo, I like that idea. We can certainly add a fn.reduce, it would operate much like a normal js reduce though, running over the full set each time. To compose a truly incremental reduce needs a few more operators. I wander if a fn.pipe() that let you compose D2 operators on a query... although that doesnt solve the type issue you have.

xenial-blackOP•11h ago

it could! I was able to compose D2 to do narrowing like this:

const pings = this.input.pipe(
  filter((e) => e.type === 'PingAdded' && e.version === 1),
  map<Event, PingAddedV1>((e) => e as PingAddedV1)
);

const pings = this.input.pipe(
  filter((e) => e.type === 'PingAdded' && e.version === 1),
  map<Event, PingAddedV1>((e) => e as PingAddedV1)
);

(y'all probably deal with enough issues that you might not have made the connection but I'm the same person who was asking about d2ts on the Electric Discord earlier this week.) BTW I ended up getting around the type mapping problem in d2ts by having one kind of stream for each event and then concatenating them together as necessary. Just had an entrypoint function that pushed events into the right input. Don't think that'd work for db, though, at least not directly. would y'all be open to a PR then @samwillis? Gotta get my start on Hacktoberfest somehow 😆

wise-white•10h ago

So yeah pipe would be even more flexible (and cheaper as still part of the pipeline)

xenial-blackOP•10h ago

I mean, would y'all be up for a PR for that then? I've been reading through the code and I'd like to take a stab at it. I don't want to waste my effort and your time on review if it needs more design consideration though. I know how sketchy it can be to add escape hatches early in an API's life

wise-white•10h ago

I'll let Sam say for realz but it makes sense to me to have an escape hatch for extending the core logic. We do have a lot going on but if you wanted to build something for everyone to look at, that'd at least help inform the final design/implementation

deep-jade•9h ago

Just want to check which option you are thinking of having a go at, the reduce, the pipe or the types? (sorry there is a lot of ideas in this thread) I'm very open to a PR on all of them. Best to sketch stuff out and open PRs early for feedback on new features.

xenial-blackOP•9h ago

pipes, probably I figure it's a pretty general solution. If you wanted it to be more specific to reductions, that's also doable.

deep-jade•9h ago

Nice! I've had that in the back of my mind since the start. Will be really nice to expose the db-ivm (d2) operators as an option.

xenial-blackOP•9h ago

(Although, I mean… every collection operation can be phrased in terms of reduce so I guess they're equivalently broad. But I think pipes would be nicer since it maintains visibility to the D2)

deep-jade•9h ago

If while working on it you find a solution to type narrowing int he pipeline id bee very happy too! We may want to put it on a different namespace on the query builder than fn as they aren't really js operators.

xenial-blackOP•9h ago

oh! I think we had different call ideas. - I thought it would be q.pipe(map(…)).pipe(iterate(…)) - It sounds like you are thinking more like q.pipe.map(…).pipe.iterate(…) Is that right? I suppose q.pipe(map(…), iterate(…)) would also be possible, and maybe natural if you're used to writing d2ts/db-ivm already (but those have been out since… May, right? Not a lot of long-held knowledge maybe.)

deep-jade•9h ago

Yep the first - so just the same api as the actual d2 pipe api. But was thinking we would place it under namespace to mark it as advanced... but that's probably not needed..

xenial-blackOP•3h ago

I'll see what I can do in terms of implementation and then we'll have a ✨ bikeshed party ✨ I had the chance to look through the guts tonight. Seems like we might want to make select and pipe mutually exclusive. Does that sound right? It’s going to be confusing which runs first/second otherwise

Gaming

Programming

typing in helpers in db-ivm

Did you find this page helpful?