Shorthand way to pass the entire update object inside of doUpdateSet when updating multiple values.

Ddecho4/18/2023
Greetings!

Thanks for developing Kysely, some of the stuff that it does actually blows my mind. I only started using it a few days ago, and I love it so far. Anyway, I have a quick and simple question.

I am trying to update (upsert ) multiple values into a table like this:

const countries = [
    { id: 1, name: 'UK', continent: 'Europe' },
    { id: 2, name: 'France', continent: 'Europe' }
];

await db.insertInto('countries')
    .values(countries)
    .onConflict(oc => oc
        .column('id')
        .doUpdateSet(eb => ({
            name: eb.ref('excluded.name'),
            continent: eb.ref('excluded.continent')
        }))
    )
    .execute();


This all works as expected, however, imagine that my countries had tons of keys besides id, name and continent. It would become really tedious and error prone if I had to write down my_column: eb.ref('excluded.my_coulmn') inside of the doUpdateSet method.

This is not an issue if you are upserting a single object at a time (like a single country instead of an entire array), because then you can just reference that object inside the doUpdateSet, but I am working with an array of objects (countries).

So my question is if there is a workaround/solution for this problem.

Cheers!
KNKristian Notari4/19/2023
I have created an helper that given an object return the "update object" with all the columns with the excluded.column thing:

export const updateAllSet = <DB, TB extends keyof DB, V extends Insertable<DB[TB]>>(
    record: V
): UpdateObject<OnConflictDatabase<DB, TB>, OnConflictTables<TB>, OnConflictTables<TB>> =>
    // eslint-disable-next-line @typescript-eslint/consistent-type-assertions
    pipe(
        record,
        R.mapWithIndex(
            column => (eb: ExpressionBuilder<OnConflictDatabase<DB, TB>, OnConflictTables<TB>>) =>
                // eslint-disable-next-line @typescript-eslint/consistent-type-assertions
                eb.ref(`excluded.${column}` as StringReference<OnConflictDatabase<DB, TB>, OnConflictTables<TB>>)
        )
    ) as UpdateObject<OnConflictDatabase<DB, TB>, OnConflictTables<TB>, OnConflictTables<TB>>


I couldn't make it completely type safe though (notice the various as). Here R.mapWithIndex is the same as mapping over all the entries of the record given as input so...
KNKristian Notari4/19/2023
I couple that utility with a chunking utility, because when dealing with large arrays (hence parameters you pass to database) there's a parameters limit (using postgres for example) so I actually chunk the array and do multiple inserts as needed. That way, I get chunks where the first elements of the array is always there, and I use that as a reference for the "updateAllSet"
Ddecho4/19/2023
Hey mate, thanks a lot for this, I really appreciate the time you spent to help.

That being said, before replying here I wanted to analyse and test your code but I'm getting some errors. The types I import from kysely, but where are pipe and R coming from?
Ddecho4/19/2023
I am kinda familiar with what piping is from linux command line, but sure exactly sure how it's implemented in your example
KNKristian Notari4/19/2023
yeah those comes from fp-ts which is another library for functional-like programming/composing in ts. You can omit those and just Object.entries . map . Object.fromEntries your record
KNKristian Notari4/19/2023
or whatever you'd like to map over the record keys/objects
Ddecho4/19/2023
right, i figured they might come from a popular lib
Ddecho4/19/2023
and then the implementation would be like:

await db.insertInto('countries')
    .values([{ id: 1, name: 'France', continent: 'Europe' }])
    .onConflict(oc => updateAllSet(oc))
    .execute();
KNKristian Notari4/19/2023
not really. You need to prepare your onConflict how you want, then call oc.doUpdateSet(updateAllSet(record)) where record is the first element of your values
KNKristian Notari4/19/2023
(I use the first one usually, don't know if they change in your case)
Ddecho4/19/2023
okay yes, that makes more sense
Ddecho4/19/2023
await db.insertInto('countries')
    .values([{ id: 1, name: 'France', continent: 'Europe' }])
    .onConflict(oc => oc
        .column('id')
        .doUpdateSet(updateAllSet(oc))
    )
    .execute();
Ddecho4/19/2023
ok, thanks for showing me this helper, need to do some brainstorming and analysing, only started a few days ago with kysely
KNKristian Notari4/19/2023
const values = [{ id: 1, name: 'France', continent: 'Europe' }]
await db.insertInto('countries')
    .values(values)
    .onConflict(oc => oc
        .column('id')
        .doUpdateSet(updateAllSet(values[0]))
    )
    .execute();
KNKristian Notari4/19/2023
more like this
KNKristian Notari4/19/2023
it's up to you to check of the first value not being undefined (eg. being there)
Ddecho4/19/2023
ok seems like i misunderstood what the helper does then
Ddecho4/19/2023
it's point is just to extract the column names i guess
Ddecho4/19/2023
hey I'm sorry to bother but could you show me the exact imports you're using from fp-ts. Seems like there is a ton going on with that library and I'm familiar with it
KNKristian Notari4/19/2023
you can use the helper without any other libraries if you want
KNKristian Notari4/19/2023
the "helper" is just a mapper for each key value pair of one of the values to set the eb.ref('excluded.${column}')
KNKristian Notari4/19/2023
const record = { a: "hello", b: 42 }
const updateAllSet = (r) => {
   const entries = Object.entries(r)
   const mapped = entries.map(([k,v]) => [k, (eb) => eb.ref('excluded.${k}')])
   return Object.fromEntries(mapped) 
}
Ddecho4/19/2023
okay, thanks that would be a better starting point since i was struggling to understand exactly what was going on in the previous function
Ddecho4/19/2023
yep ok that worked minus all the typescript errors, cheers 👍
Ddecho4/19/2023
i will figure the rest out
KNKristian Notari4/19/2023
😉
Ddecho4/20/2023
Well, after studying your example and brainstorming quite a bit, I came up with my modified version:

function updateAllSet<DB, TB extends keyof DB, V extends Array<Insertable<DB[TB]>>>(records: V) {
    return Object.keys(records[0])
        .reduce<UpdateObject<OnConflictDatabase<DB, TB>, OnConflictTables<TB>, OnConflictTables<TB>>>((acc, key) => {
            return {
                ...acc,
                [key]: (eb: ExpressionBuilder<OnConflictDatabase<DB, TB>, OnConflictTables<TB>>) =>
                    eb.ref(`excluded.${key as string}` as StringReference<OnConflictDatabase<DB, TB>, OnConflictTables<TB>>)
            };
        }, {});
}


However, I can't say I'm really satisfied. On runtime level, I understand everything. On compile/type level, not so much, especially some of these UpdateObjects with tons of generic params, really difficult to trace exactly what's going on.

It would have been so much easier if ref provided some extra helpers, like:

doUpdateSet(eb.ref.allExcluded()))
// or
doUpdateSet((eb) => eb('excluded'))


something like this.

But I guess this solution also isn't bad and you gotta work with what you got.
KNKristian Notari4/20/2023
Kysely doesn’t store a runtime value defining all the possible columns. It should rely on the values you passed in. But what if you are doing an insert from other tables, so with an expression method instead of values? It’s hard to get it right. Also using the first value or the last one could, potentially, be different, based on the array of values you gave in. Even if the type is respected, they could have different keys, so you could end up creating an exclude update for a non existing column at db. If you need a subset of excludes you can write that manually, key after key. If you want to mentally say “everything I gave you as values” or “everything this table specifies as columns” you should do the values[0] thing I’m doing (if you can trust the keys in the values you’re using) or create a dummy “table record” with the keys on the runtime/value level, so you can use that object as the record to pass to the helper I wrote to you
KNKristian Notari4/20/2023
Either way, I advice using/creating a plugin for kysely (you would like to write your own) which filters the non existing columns from stuff like insert/update or on conflict set as in your example. Cause you don’t really know what could end up being in the objects you pass as values to kysely at runtime, even if the type is correct they could have more keys which map to non existing columns. For kysely/this plugin to do so you have to rely on a dummy table record value built just for this purpose
Ddecho4/20/2023
So what you're basically saying is, what if we do a join from another table (or some other custom logic) and then we want to magically remap all to ref('excluded....').

Kysely and by extension ref would usually have the type information about that new data, but such data doesn't exist in runtime, so it's not possible to magically/automatically remap that on runtime level.

You're right I think that explanation makes a whole lot of sense.
Ddecho4/20/2023
for the time being I'm using this method which should be safe enough:

import type { Countries } from '../types/index.js'; // kysely-codegen
type Country = Insertable<Countries>;

const data = [{ id: 1, name: 'UK' },{ id: 2, name: 'France' }];

const countries = data.map<Country>(country => ({
    id: country.id,
    name: country.name
}));

// proceed to do database stuff with "countries"

but once I get my hands more dirty and gain better understanding of kysely I will most likely consider your suggestion.
Ddecho4/20/2023
by the way if you don't mind me asking, are you also a developer/contributor to kysely or just like helping people on the server?
Ddecho4/20/2023
I have to say huge thanks again 🙂
KNKristian Notari4/20/2023
what I've done for one of my codebases was to create a plugin which accepted all the database tables with key: true values like:
type Database = {
  a: {
     a1: string
     a2: number
  },
  b: {
     b1: string
     b2: number
  }
}
const Database = {
  a: {
     a1: true,
     a2: true
  },
  b: {
     b1: true,
     b2: true
  }
}

and then I initialize such plugin with the Database value. Doing so I can filter out unexpected keys when doing stuff like insert from values or on conflict excluded update set
KNKristian Notari4/20/2023
just helping and loving kysely approach so far
Ddecho4/20/2023
so basically you're creating a runtime representation of your types, that's a cool idea i might steal
Ddecho4/20/2023
😉
KNKristian Notari4/20/2023
yeah, so then you can use your values[0] as a record to base your eb => excluded thing for update set on conflicts without worrying about unexpected columns
Ddecho4/20/2023
then you can import and use that for dummy
Ddecho4/20/2023
great idea
KNKristian Notari4/20/2023
IMO less to reason about, more time spent on actual business logic
Ddecho4/20/2023
me too, initially I wanted to use Prisma, but I tested it and it doesn't even do joins properly. like it does multiple separate queries/calls to the database and glues them together in javascript land
Ddecho4/20/2023
performance obviously isn't going to be as good
Ddecho4/20/2023
but yeah im glad i found this amazing library
Ddecho4/20/2023
just need to get some better understanding of all the complex types, but that will probably come with experience
Ddecho4/20/2023
if you have the privileges, please tag the thread as "solved", for some reason I can't do it myself
Ddecho4/20/2023
thanks again
KNKristian Notari4/20/2023
@decho have you followed this ?
Ddecho4/20/2023
I read it yesterday but didn't pay enough attention and was trying to manually tag my thread 😅
cheers mate. if you don't mind though, I will tag my own solution instead of your own because it's using 3rd party libs and stuff so it would be easier for someone coming later if they can see native JS methods.

but we both know that i was never gonna reach that point without your help
Ddecho4/24/2023
Hey, @Kristian Notari you mentioned you create runtime objects based on your DB types. if you are using kysely-codegen I made a small script that reads the file it generates, and then generates another file file with the runtime object like this:

/** Generated by ts-morph script do not edit. */

import type { DB } from './index.js';

export type DBDummy<T = DB> = {
  readonly [K in keyof T]: ReadonlyArray<keyof T[K]>;
}

export const dbDummy: DBDummy = {
  countries: [
    'id',
    'name'
  ],
}


Still a work in progress but perhaps it can be useful to you.

https://gist.github.com/virtuallyunknown/bdb68925ab2a40c0e4e07ba8ca30741e
KNKristian Notari4/24/2023
Thanks for sharing! Right now I’m doing it manually cause it’s typed to be something specific for an input to one of my plugins and I get compile time errors when I add a column or a table and that’s not reflected in the runtime values for the plugin, so I guess I don’t really need it for my usecase, but still helpful
Ddecho4/24/2023
no problem, I thought I'd share with you just in case you might find it useful, perhaps for some future project or something like that
Ddecho4/24/2023
btw I just finished putting everything together, the updateAllSet function with your help from a few days ago, and making that work in conjunction with the runtime types generator script.

I'd be interested to hear your opinion/feedback about it, but I don't want to become annoying so if you want to take a look I can paste the code, but otherwise no probs.
KNKristian Notari4/24/2023
Sure go ahead
Ddecho4/25/2023
Well basically you have these 2 generated files, one by kysely-codegen, and the second by my script I just showed you:

// db.ts
export interface Countries {
  id: number;
  name: string;
}
export interface DB {
  countries: Countries;
}

// db-runtime.ts
import type { DB } from './db.js';

type DBDummy = {
  readonly [K in keyof DB]: ReadonlyArray<keyof DB[K]>;
}
export const dbDummy: DBDummy = {
  countries: [
    'id',
    'name'
  ]
}


then the function:

// kysely.ts
import type { OnConflictDatabase, OnConflictTables, StringReference, ExpressionBuilder, UpdateObject } from 'kysely';
import type { DB as PGDB } from '../types/db.js';
import { dbDummy } from '../types/db-runtime.js';

export function updateAllSet<DB extends PGDB, TB extends keyof DB & keyof PGDB>(table: TB) {
    return dbDummy[table].reduce<UpdateObject<OnConflictDatabase<DB, TB>, OnConflictTables<TB>, OnConflictTables<TB>>>((acc, column) => {
        return {
            ...acc,
            [column]: (eb: ExpressionBuilder<OnConflictDatabase<DB, TB>, OnConflictTables<TB>>) =>
                eb.ref(`excluded.${column as string}` as StringReference<OnConflictDatabase<DB, TB>, OnConflictTables<TB>>)
        };
    }, {});
}


finally, you use it like this:

import { db } from '../util/db.js';
import { updateAllSet } from '../util/kysely.js';

const countries = [{ id: 1, name: 'a' }, { id: 2, name: 'b' }];

await db.insertInto('countries').values(countries)
    .onConflict(oc => oc
        .column('id')
        .doUpdateSet(
            updateAllSet('countries') // here you get type safety and can only select "countries"
        ))
    .execute();


And that's basically it.
Ddecho4/25/2023
Overall I am happy with how that turned out.
KNKristian Notari4/25/2023
Ok so basically you can get rid of passing an actual record to updateAllSet
Ddecho4/25/2023
yes pretty much. and since we have the column names as an array/tuple in runtime, we just use reduce on them to output a mapped object with eb.ref('excluded...') for each one of the array members.

the only part i was slightly unhappy about was the generics in the function signature, i had to do cheat a little bit

1. TB extends keyof DB & keyof PGDB
2. use spread operator inside instead of acc[key] = eb.ref(...)

otherwise Typescript would just complain that table: TB can't be used to index dbTable.
Ddecho4/25/2023
but oh well, it works