genetic-orange

typing in helpers in db-ivm

Hey! I'm using db-ivm to aggregate events from an event-sourced system for display. Working great so far! I've noticed a lot of boilerplate around sorting and filtering, though, and so I've tried to make myself a helper:

export function build<Event extends { timestamp: number }, Out, Key extends string | number>(
  builder: (events: Event[]) => Out | null | undefined,
) {
  return reduce<Key, Event, Out, Keyed<Key, Event>>((events: [Event, number][]) => {
    const res = builder(
      events
        .filter(([_, mult]) => mult === 1)
        .map(([ev, _]) => ev)
        .toSorted((a, b) => a.timestamp - b.timestamp),
    )

    if (res) {
      return [[res, 1]]
    } else {
      return []
    }
  })
}

export function build<Event extends { timestamp: number }, Out, Key extends string | number>(
  builder: (events: Event[]) => Out | null | undefined,
) {
  return reduce<Key, Event, Out, Keyed<Key, Event>>((events: [Event, number][]) => {
    const res = builder(
      events
        .filter(([_, mult]) => mult === 1)
        .map(([ev, _]) => ev)
        .toSorted((a, b) => a.timestamp - b.timestamp),
    )

    if (res) {
      return [[res, 1]]
    } else {
      return []
    }
  })
}

This works fine, except that I have to manually specify the type parameters or they're inferred to be never. For example, the type params in this call are required:

build<
  QuestionTitleSetV1 | QuestionAddedV1,
  { title: string; type: QuestionAddedV1['data']['type'] },
  string
>((events) => {
  // …
})

build<
  QuestionTitleSetV1 | QuestionAddedV1,
  { title: string; type: QuestionAddedV1['data']['type'] },
  string
>((events) => {
  // …
})

I see an addOperator call to graph—is that a better way to add helpers like this? Otherwise, any tips on getting type inference working correctly?

33 Replies

jolly-crimson•2mo ago

curious why not just use DB directly? db-ivm is very flexible so hard to tame down the complexity which is why DB exists (in part) so it can generate really precise/fast ivm code

genetic-orangeOP•2mo ago

Can it make aggregates out of events? I didn’t see anything like that in the docs. It seemed pretty record-oriented

jolly-crimson•2mo ago

https://tanstack.com/db/latest/docs/guides/live-queries#groupby-and-aggregations

Live Queries | TanStack DB Docs

TanStack DB Live Queries TanStack DB provides a powerful, type-safe query system that allows you to fetch, filter, transform, and aggregate data from collections using a SQL-like fluent API. All queri...

jolly-crimson•2mo ago

let us know if something is missing!

genetic-orangeOP•2mo ago

Sounds like I need to read up! Thanks for the pointer So overall it seems like this package assumes you want writes to sync immediately. I’m designing this for the write authority to be on the client, though. The events are stored in IndexedDB and really only stored in Electric for browser resets and device syncing. Is there some pattern that you think would work well in db to handle this?

jolly-crimson•2mo ago

you can "persist" the data to indexedb like https://tanstack.com/db/latest/docs/overview#localstoragecollection actually would love PRing an indexeddb version of that then store txs as the events to sync around

genetic-orangeOP•2mo ago

would you mind expanding on the last part, actually? do you mean storing sync state alongside the events? I guess then I’d make manual transactions based on that data?

jolly-crimson•2mo ago

Yeah, create a custom transaction type that stores the txs when committed

genetic-orangeOP•2mo ago

hmm, I'll look at that. Thanks. ok, so I tried this out @Kyle Mathews and… is there some aggregation you had in mind here? I don't see a way to reduce a group in the db API 🤔

export const pingEventCollection = createLiveQueryCollection({
  id: 'ping-events',
  query: (q) =>
    q
      .from({ event: eventCollection })
      .where(({ event }) => eq(event.type, 'PingAdded'))
      .groupBy(({ event }) => event.data.time)
      // what goes here to reduce?
})

export const pingEventCollection = createLiveQueryCollection({
  id: 'ping-events',
  query: (q) =>
    q
      .from({ event: eventCollection })
      .where(({ event }) => eq(event.type, 'PingAdded'))
      .groupBy(({ event }) => event.data.time)
      // what goes here to reduce?
})

It looks like .fn.select only works on a single row at a time (sure) and there's no custom aggregation… is there?

jolly-crimson•2mo ago

You see the aggregation functions?

jolly-crimson•2mo ago

https://tanstack.com/db/latest/docs/guides/live-queries#aggregate-functions

Live Queries | TanStack DB Docs

genetic-orangeOP•2mo ago

yes, I read that. But I don't think those fit what I'm after. I'm working with event data here, and the aggregates are actually a custom reduction. Is there an aggregate function where I can run some function that just gets a group and returns whatever? lemme give an example. I have these two events (simplified):

type AddQuestion = { type: "AddQuestion", data: { id: string, type: "tag" | "number" } }
type SetQuestionName = { type: "SetQuestionName", data: { id: string, name: string } }

type AddQuestion = { type: "AddQuestion", data: { id: string, type: "tag" | "number" } }
type SetQuestionName = { type: "SetQuestionName", data: { id: string, name: string } }

and I want to produce an aggregate that looks like this:

type Question = {
  type: "tag" | "number"
  name: string
}

type Question = {
  type: "tag" | "number"
  name: string
}

I can where and then groupBy (subject to ignoring an error about type narrowing) but in order to produce the desired output type I need something like db-ivm's reduce. that's why I was using that instead of db directly in the first place. 😅

genetic-orangeOP•2mo ago

basically if I could make a custom one of these I think I'd be in business. https://github.com/TanStack/db/blob/c4c2399cbe0969da01714ef59f4aa05ffdcb75dd/packages/db-ivm/src/operators/groupBy.ts#L147-L163

GitHub

db/packages/db-ivm/src/operators/groupBy.ts at c4c2399cbe0969da0171...

A reactive client store for building super fast apps - TanStack/db

jolly-crimson•2mo ago

cc @samwillis seems like a fn.reduce function would be useful

genetic-orangeOP•2mo ago

if a hypothetical fn.reduce got a signature like (in: T[]) => U[] I think it might also be an escape hatch for type narrowing like we were talking about in the other thread.

quickest-silver•2mo ago

Ooo, I like that idea. We can certainly add a fn.reduce, it would operate much like a normal js reduce though, running over the full set each time. To compose a truly incremental reduce needs a few more operators. I wander if a fn.pipe() that let you compose D2 operators on a query... although that doesnt solve the type issue you have.

genetic-orangeOP•2mo ago

it could! I was able to compose D2 to do narrowing like this:

const pings = this.input.pipe(
  filter((e) => e.type === 'PingAdded' && e.version === 1),
  map<Event, PingAddedV1>((e) => e as PingAddedV1)
);

const pings = this.input.pipe(
  filter((e) => e.type === 'PingAdded' && e.version === 1),
  map<Event, PingAddedV1>((e) => e as PingAddedV1)
);

(y'all probably deal with enough issues that you might not have made the connection but I'm the same person who was asking about d2ts on the Electric Discord earlier this week.) BTW I ended up getting around the type mapping problem in d2ts by having one kind of stream for each event and then concatenating them together as necessary. Just had an entrypoint function that pushed events into the right input. Don't think that'd work for db, though, at least not directly. would y'all be open to a PR then @samwillis? Gotta get my start on Hacktoberfest somehow 😆

jolly-crimson•2mo ago

So yeah pipe would be even more flexible (and cheaper as still part of the pipeline)

genetic-orangeOP•2mo ago

I mean, would y'all be up for a PR for that then? I've been reading through the code and I'd like to take a stab at it. I don't want to waste my effort and your time on review if it needs more design consideration though. I know how sketchy it can be to add escape hatches early in an API's life

jolly-crimson•2mo ago

I'll let Sam say for realz but it makes sense to me to have an escape hatch for extending the core logic. We do have a lot going on but if you wanted to build something for everyone to look at, that'd at least help inform the final design/implementation

quickest-silver•2mo ago

Just want to check which option you are thinking of having a go at, the reduce, the pipe or the types? (sorry there is a lot of ideas in this thread) I'm very open to a PR on all of them. Best to sketch stuff out and open PRs early for feedback on new features.

genetic-orangeOP•2mo ago

pipes, probably I figure it's a pretty general solution. If you wanted it to be more specific to reductions, that's also doable.

quickest-silver•2mo ago

Nice! I've had that in the back of my mind since the start. Will be really nice to expose the db-ivm (d2) operators as an option.

genetic-orangeOP•2mo ago

(Although, I mean… every collection operation can be phrased in terms of reduce so I guess they're equivalently broad. But I think pipes would be nicer since it maintains visibility to the D2)

quickest-silver•2mo ago

If while working on it you find a solution to type narrowing int he pipeline id bee very happy too! We may want to put it on a different namespace on the query builder than fn as they aren't really js operators.

genetic-orangeOP•2mo ago

oh! I think we had different call ideas. - I thought it would be q.pipe(map(…)).pipe(iterate(…)) - It sounds like you are thinking more like q.pipe.map(…).pipe.iterate(…) Is that right? I suppose q.pipe(map(…), iterate(…)) would also be possible, and maybe natural if you're used to writing d2ts/db-ivm already (but those have been out since… May, right? Not a lot of long-held knowledge maybe.)

quickest-silver•2mo ago

Yep the first - so just the same api as the actual d2 pipe api. But was thinking we would place it under namespace to mark it as advanced... but that's probably not needed..

genetic-orangeOP•2mo ago

I'll see what I can do in terms of implementation and then we'll have a ✨ bikeshed party ✨ I had the chance to look through the guts tonight. Seems like we might want to make select and pipe mutually exclusive. Does that sound right? It’s going to be confusing which runs first/second otherwise

quickest-silver•2mo ago

I'm not sure, groupby is dependent on select, so would also be excluded. That's probably ok. I think pipe running after select is an option, the complexity comes from ensuring that the output of the pipe is materializeable - needs to be a keyed stream.

genetic-orangeOP•2mo ago

so… I've messed around with this a bunch and realized that I truly do just want to use db-ivm. I've got some specific ways I want to manage local data (plus combining multiple sources) that make db-ivm a better fit than db. As a result, back to the original question: how would you get the types inferred here?

quickest-silver•2mo ago

I'll need to go back and look how the types on this all works. (DB mostly sidesteps the inferred types on this) There are a few operators that are composed from others, and I think I made the type infer on them. Take a look how they work. Maybe filterBy: https://github.com/TanStack/db/blob/main/packages/db-ivm/src/operators/filterBy.ts

GitHub

db/packages/db-ivm/src/operators/filterBy.ts at main · TanStack/db

A reactive client store for building super fast apps - TanStack/db

quickest-silver•2mo ago

I think you also asked about type narrowing in a filter, I'm not sure what the best option there is... it would be nice to find a way.

genetic-orangeOP•2mo ago

I think mapping via a schema would be one way. It would have a runtime cost but since you have to do the check anyway… filterBy looks helpful. Thanks for that just to close this out for anyone else who finds this thread, it ended up being this:

export function build<
  In,
  InK extends In extends KeyValue<infer K, infer _V> ? K : never,
  InV extends In extends KeyValue<InK, infer V extends { timestamp: number }> ? V : never,
  Out,
>(builder: (values: Array<InV>) => Out): PipedOperator<In, KeyValue<InK, Out>> {
  return reduce((events) => {
    try {
      const res = builder(events.filter(([_, mult]) => mult === 1).map(([ev, _]) => ev as InV))

      return [[res, 1]]
    } catch (error) {
      if (error instanceof ZodError) {
        return []
      }

      throw error
    }
  })
}

export function build<
  In,
  InK extends In extends KeyValue<infer K, infer _V> ? K : never,
  InV extends In extends KeyValue<InK, infer V extends { timestamp: number }> ? V : never,
  Out,
>(builder: (values: Array<InV>) => Out): PipedOperator<In, KeyValue<InK, Out>> {
  return reduce((events) => {
    try {
      const res = builder(events.filter(([_, mult]) => mult === 1).map(([ev, _]) => ev as InV))

      return [[res, 1]]
    } catch (error) {
      if (error instanceof ZodError) {
        return []
      }

      throw error
    }
  })
}

(I do the former toSorted in a separate pipeline step now) Huh, that actually doesn’t work the way I thought it would! What I thought was a multiplicity parameter is getting set to greater than one sometimes. What is that actually used for? Is it the frontier version or something?

Gaming

Programming

typing in helpers in db-ivm

Did you find this page helpful?