Typescript build fails with large number of columns
I have a table with ~460 columns, and on trying to run typescript on my app, I'm getting "error TS2589: Type instantiation is excessively deep and possibly infinite." It goes away once I reduce the number of columns in the Drizzle mysqlTable call to <= 357. Is there any way to address this?
123 Replies
why would you have 460 columns on a table....
I can also get it to fail by adding complexity to column definitions, eg with that 357 columns it goes back to failing if I add
.notNull().primaryKey().autoincrement() on the last column
Not by choice! Legacy DB schema
It should really be taking advantage of JSON columns as it's relatively sparse, but this is coming from a 10 year old application that was originally running on like mysql 5.5 and Drupal...
NB: This is storing experimental data from a research project, where various experiments have had a lot of different types of data points
I don't want to migrate it to something different now, as I'm trying to keep things backwards compatible with our legacy systemHonestly, I'm not sure if it even can be resolved at the current stage - the type inference is simply too much for TypeScript to handle. Theoretically, we could make it easier for TS by asserting some intermediate types, but that's not implemented in the current API.
What you can TRY to do is describe it partially with views
But that will still only work for selects, since we don't have anything else for views
Strictly speaking, I can just omit this model since I've been using some lightweight raw SQL anyways, but this still seems like a problematic failure mode
IE, if you introspect a large table, you're just left with a broken project
How did you work with this table before you tried Drizzle?
Even with ~200 columns, intellisense grinds to a halt
And I know that there are schemas in eg enterprise software that get that large
How did you work with this table before you tried Drizzle?Raw SQL in PHP 🙂 And in the more recent term, Prisma (works fine, though they do a lot more pre-processing in their types)
Yes, they basically codegen all the types and the client
Whereas we do everything via inferring, which puts a lot more pressure on TS
Looking at the way the Drizzle schema builder is set up, it seems like a looooot of layers of indirection
yeah, the types are pretty complex to infer everything properly
eg looking at a column definition, there's 7 exports
Though I guess a bunch of that is builder alias vs builder vs concrete class and config types
Let me try to test some simple type assertion API and check if it'll help
I'd be interested to see additional details about how the types are currently intended to work - looking at it now, I'm getting a bit lost
Specifically column and column builder
Hmm, I've tried building a table with 500 columns, and not seeing the error. Which column types are you using?
I've tried using 500 int columns
double
For most of them
And a couple dozen varchar, int, and bigint
What ts version are you using btw? This project is still on 4.9.5, maybe there's been improvements if you're using 5.x?
5.0.3
Well, I changed a bunch of columns to varchar, int and bigint, still no error
I now have 1000 columns in the table
Updated to ts 5 and it's no longer failing
amazing
The intellisense still takes a couple minutes to kick in
But it at least doesn't outright error
Well, now it depends on how powerful your PC is 🙂
I'd be curious to know what part of this is actually causing the issue
I think TypeScript provides some tracing utilities
For type checking
phew - going from ~10 columns to ~250 columns, the diagnostics show something like an additional 200,000 types being created
Wondering if this comment is relevant? https://github.com/microsoft/TypeScript/issues/39678#issuecomment-663212548
GitHub
Slow compilation likely caused by too many types · Issue #39678 · m...
TypeScript Version: >= 3.7.5 (reproduceable in @latest and @next) Search Terms: slow compilation typecheck types typecount After refactoring the model of our project to rely more heavily in clas...
GitHub
Slow compilation likely caused by too many types · Issue #39678 · m...
TypeScript Version: >= 3.7.5 (reproduceable in @latest and @next) Search Terms: slow compilation typecheck types typecount After refactoring the model of our project to rely more heavily in clas...
hmm, interesting
we do use Pick and Omit in some places
One thing that seems very useful to note
If all the columns are named identically, there is no type explosion
This is fine
This is not
I assume typescript would normally de-duplicate types, but because this changes each column type, it can't do any short-circuiting
Looks like when the columns are identical, it adds ~4 types. When I change the name, it adds ~888 types
The added types also show up when adding columns to different tables. Presumably the error I was getting in 4.9 had something to do with the number of types that had to be evaluated on a singular object, but the slowdown issue still winds up being an issue when taking into account the total number of columns in your schema across all tables
With some ~400 colums across 50 models, it takes ~10 seconds for intellisense to kick in after a change. ~350k types. 26 seconds and ~425k types when I have ~450 split across two models.
That's a lot. I'll add an item to backlog to investigate the performance issues. Thanks for the report!
I'm wondering, is the original column name in the generated types actually used anywhere? If not, just making it
string instead of extending from string would resolve a large majority of the issuesyou mean the DB column name? yes, it's used for Knex/Kysely adapters, since those require DB names
Ah!
had to specifically add it for them 🙂
Makes sense
There's a lot of type transformation going on in the column builder, when you invoke the chain methods
I'll probably need to look into it in the first place
The current iteration of those types is a balance between the internal types convenience and the load on TS compiler
But the most heavy type magic I think happens in joins
It's easily the most complex place in the whole ORM
BTW what does HKT stand for? I keep seeing it in the type but it's not clear what it stands for
higher-kinded types
a concept from FP that's not yet properly implemented in TS
in our case, it's used to allow changing the generics on a certain type without knowing what exactly type that is
so that if you have a PgIntegerBuilder and you invoke .notNull on it, which is defined way down on the base ColumnBuilder class, you'll be able to change the generics and still keep the type as PgIntegerBuilder and not ColumnBuilder
That's a good example
I get the idea in theory, but the amount of things going on here admittedly make my head spin a little 🙂
same here, and I wrote the thing
😄
I have to wonder, how much simpler would this be if you just used class instances instead of the builders?
but builders are class instances
you just create them via calling a function instead of
new
if you check the code for any one of them, all they do is create a class instance and return itRight, but you have to construct the resulting type based on chained methods, vs a single consturctor parameter
ah I see, so get rid of chaining
yes, that might simplify the types, but it'll make the API look worse IMO
Yeah I agree the chaining is elegant
chaining looks much nicer and resembles SQL
we also have plans to get rid of the required DB name as 1st argument, which will make it look even nicer
Yes, I saw that
fwiw, I think it matters less in the schema definition than it does in the queries themselves
true
you write your schema once
but it's still nice
I don't disagree!
schema definition is the 1st thing you do, it's like a first impression
On the same token:
- What is the readability or usability benefit of
.notNull() vs {notNull: true}?
- Wouldn’t it be significantly easier to maintain with a non-builder config, since you could go from ~7 classes/interfaces per column to ~2 and avoid a large amount of type operations?
- I imagine the perf issues would be largely solved due to the significantly lowered number of intermediate types
That said, I realize I’m a bit out of my depth not having a deep knowledge of the implementation, so take it with a grain of salt
Spent a little time trying to work through it myself to see if I could simplify with the builder pattern, and I'm following a bit more why it's as complex as it is. Unfortunate price for polymorphic builders that modify types
While I don’t find the tradeoff worthwhile, ultimately it’s not up to me. Could probably be improved with some additional docsThe main reason for the chain syntax was SQL-likeness
integer('id').notNull().default(0) -> id integer not null default 0Vs
{ name: 'id', notNull: true, default: 0 } I guess it is slightlyGood point
Another thing that the chain syntax allows is to modify the column builder in different places
For example, create it in some factory and further customize in other place
In that case you’d probably rely on spread syntax or such
But maybe slightly less ergonomic
Not really, since then you'll have to pass those additional properties into the factory
instead of calling them after the factory returned the result
Not necessarily
Consider:
Where getBaseProps is typed to return IntConfig
ok, this is the "props factory" then, instead of the column builder factory
yeah, that might work
but it doesn't look as nice as the chain syntax, still 🙂
As I said, it’s ultimately not up to me
But it seems like you’re adding a lot of complexity for a marginal readability benefit
well, it's all different parts of DX
both the syntax and type complexity
Yep
thanks for the suggestions though! it's always good to get a side opinion.
I personally find the props version just as usable, but that’s just my opinion
I’ve been trying to play to see if there’s a way to get the builder implementation to be less complex, but building types with polymorphic builders is really hard
yep
As I’m sure you well know. My hubris thought otherwise lol
we might consider improving things there, but currently have more important stuff to implement
Sure
@bloberenober Had some shower thoughts about this, and came up with an alternate approach for the builder. I'm not fully convinced if it's cleaner or not, and it may be missing some features you have in the current implementation, but wanted to shoot it by you and get your thoughts https://www.typescriptlang.org/play?#code/JYOwLgpgTgZghgYwgAgMIHsA2BXAtiDEGYAc2QG8AoZG5EdMAOW00wC5kAjdLCOESgF9KlUJFiIUGHPgBC2YJgAm0ADyFiZCAA9IIJQGc0WPAXRFSyALzGZZiyQB8FarQTnNHDaQDcQkTDYIAhgwObIuHAA1hDSpvKKKlCqCcrQyDp6hrbxCmlQjgAUnHlJHKlJAJQutMhQEGDYUCA1tbQAdJ0lidAANK5tdAzMrIXVVIOD7iAGYBTTmr3Ine0GEJgwgtbIYAAWwAZ+k23TsxT0TCyYSyv1kaCgJN4kWzYLviLHtfWNzRHRsRMclK0EK5BWaw2S3eJA44M6dzgDxATw8pCWFxG7B2UGwEEEgkqAzawlqwmElECwVC4XcdlUAFFdFA4M8ipkWc8OEywJy0SRxsSfk0WpEYnFgT0oGCYXDMVcOPBMGsbp0Oaz+QTKn4KWJoPAkMgAGpwKAIXamiX2TQZXQQfRGK3PVo0TD2kh7DggPCcaA6kR6iSGk1mi1QK0VNTOzL27Ih82WoHWyw2eNhp385wxh05SX5dSZigUqkhMKigFpxN2SPJGu2rJGSvhpM1ordfLlEFQQXfBoil1tFbtpL9Y5ulF7QogCAAdwAMu7PXQfdAe18aKc5uQYarVutNts9gcjuvkJuKOOPbtd4jkaiHK8z-y-MTjsK-mKIE2I12wRD99C-JwrcED3CAjzPEsl5LtO86LrsWqvrQpLIf4lJBKW4QAG6mgm0prjQ77ljE34tr+dKmGC0G7BwAAMhLamhbpzPUBiKnAyooDYOGhqaYztDC7TyqwfjMXUEAGAAjBwvJ4tsPF4fxwmYPxgnKT4QA
TS Playground - An online editor for exploring TypeScript and JavaS...
The Playground lets you write TypeScript or JavaScript online in a safe and sharable way.
It uses functions returning raw objects instead of classes to get around the need to have a polymorphic reference to the subclass
It's possible there may be a way to simplify it further, but this is as far as I got right now
wow, what a thread
Sorry, interesting technical problem got the better of me 😅
Yeah want to follow up on this again - even omitting that large DB model, intellisense is really slow once I start importing from the schema file
Like, "I hit
. and it takes 10 seconds to see results" slow. It's a huge productivity killer
What I'm doing in the meantime is when generating my schema, setting the column name type parameters to 'string'Is there a way to workaround this? I tried migrating our production code with pg that has 26 tables and 14 columns each on average, the intellisense is taking up to 5s to show, would be great if there is a temporary solution
Temporary solution is to pass
string as the generic parameter for the column nameGeneric seems hard to work for certain column type helpers (like text), but
as string seems ok. Thanks, will also try your solution.
hmmm doesn't seem to work though...
my intelisense is still very slow >.<"
ok, i tried switching to webstorm from vscode, the intellisense is by worst up to ~1.5s only
still a lot, but better
we will improve types to make typescript work faster with it. It's not something we are going to work right now, but it's a priority for us
my team tried out vim, instant intellisense, so it's vscode doing some stuffs behind the scene, we also tried disabling vscode extensions and re-enabled 1 by 1, it was still very slow
another point to vim
I wish I could use it without breaking my fingers and remembering which button to press
oh, that's actually great feedback
here's the vim recording
amazing
wow
we can make a banner in docs
"Use Vim to make intellisense instant"
I’m incredibly surprised, I don’t know what would make it faster in VIM, I was seeing noticeable slowdowns even just with running tsc
are you also using vscode?
Ok diving deeper does allow us to understand more, it's not vim that brings the improvements, it's my team was using coc-tsserver with vim which is a fork from the official tsserver implementation, the coc-tsserver implementation seems to be faster because it doesn't have all the latest typescript features from the first look
I use Neovim with the built in LSP and code suggestions are instant, so it’s not just coc.
How many total columns do you have? How many files are they split between?
If you only have a file with 10 columns imported, it won’t be noticeable. If you wind up having a couple hundred columns in imported files, that’s when it gets to be a slog
Yep, you're right! Just tested with hundreds more columns and it takes quite a long time. Not even sure how long as I never saw anything come up 😅
did this actually work faster for u?
Kind of (much faster than before) I can't remember the last time I wait for intellisense 😅
so basically just add
as string to all the columns? does that break anything?yes (if it matters, I have a M1 pro chip, maybe it compensates?)
i'm also using m1 pro
oh crap, it really helps a lot
https://devblogs.microsoft.com/typescript/announcing-typescript-5-1/#avoiding-unnecessary-type-instantiation maybe it'll improve our TS issues?
Daniel Rosenwasser
TypeScript
Announcing TypeScript 5.1 - TypeScript
Today we’re excited to announce the release of TypeScript 5.1! If you’re not yet familiar with TypeScript, it’s a language that builds on JavaScript by adding constructs called types. These types can describe some details about our program, and can be checked by TypeScript before they’re compiled away in order to catch possible typos,
Vscode now uses TS 5.1. Maybe it's just me but it feels betters than before 😆
i've been using 5.1 by configuring vscode to use node module's one, so i couldn't feel the difference between 5.0 and 5.1...
M1 Mac Pro here, Neovim running tsserver. 49 tables. 1,000 lines in schema total so say 800ish total columns. Drizzle is completely unusable. My db type defaults to any and I get 0 type recognition. When reducing the project to a single table, it works great. I haven't analyzed exactly when the types breakdown due to scale, but the size of the schema is definitely the issue.
I have 13 tables most table with around 10-12 column
i am on windows 10, with 32 gb RAM, still facing issue with code completion, suggestions, showing ts error , sometimes ts server just stoped working need to restart VSCODE.
i am sharing the video
https://discord.com/channels/1043890932593987624/1119853271616602203
check his thread i post a video and screenshot of my pc configuration.
its really hard to maintain and work on medium label project. please fix this. its very problematic
We are aware of this issue, and it's not something that we can fix quickly. We are experimenting with types to make them less complex and help the server function more efficiently.
We are doing it meanwhile implementing new features that we see important for us and for the community
We have a few things to finish in June and needed part of a July will be spent trying to make types smaller
ok thanks for replying, but started migration our production app to drizzle , facing issue and slowing down the development.
you guys did a great job please try to fix this issue as soon as you can
thanks again for the great ORM
I understand that, and I know what it's like to wait for developers to fix something that is blocking the development flow. We've been in that situation as well. 😅
Just to clarify, we are putting in our best effort to dedicate as much time as possible to improving Drizzle, both for existing and new users. However, not every day can be entirely free after our main jobs, so there may be weeks where progress is slower, while other weeks may be faster.
Nevertheless, I greatly appreciate the support and patience from the community. It means a lot to us
yeah i also appreciate your efforts
Thanks for all your effort. I'm excited to migrate over to Drizzle when it's ready whenever that may be.
If you need a big schema to test realistic implementation, let me know
Is there any updates on this?
it's one of our priorities for this month
No pressure but I'm worried to proceed with Drizzle if the problem can't be addressed soon. Is it even fixable or does it entirely depends on TypeScript compiler? Some update would be indeed very helpful. Thank you.
This is exactly what @Dan Kochetov is going to work on now. We will update here as soon as we have some results. We had 1 important release to prepare, which should be live soon, and this is our next top-1 priority.
We are not moving forward with Drizzle features until this one is fixed, so we are fully focused on resolving it. We didn't have a chance to start working on this one earlier because we had to spend a bit more time on fixes for a release we will be dropping soon
Awesome! I'd expect you running into this issue as well with your own projects right?
yes
we simply don't need the type checks, the compiler runs directly in the mind
I have a production project with 77 tables, but it is using an old drizzle version (0.11.6) that is over 1.5 years old. While it still works really well in production, the data types have evolved significantly since then, and we haven't had a chance to upgrade it because everything was working well for us
Let me downgrade to 0.11.6 
if you want to see how it looked like in 0.11.6
https://drizzle.team/article/drizzleorm-v011-18-months-into-development/
👀
whenever got any notification from this. I just hope this issue fixed 😊
exactly
🙏
You can check types issue here
drizzle-orm@0.28.0
We've run the diagnostics on a database schema with 85 tables, 666 columns, 26 enums, 172 indexes and 133 foreign keys. We've optimized internal types which resulted in 430% speed up in IntelliSense.
https://github.com/drizzle-team/drizzle-orm/releases/tag/0.28.0Awesome! I’ll be sure to give the new version a go next week after I get back from vacation
TSC diagnostics types went from 106k to 81k and I only have about 8 tables, ~10 columns each. Good job, thanks!
just updated, and without a single change watching with tsc finishes almost instantly...this single-handedly made drizzle the obvious choice!
Nice!
My two biggest issues with drizzle fixed in one patch, way to go lol
🫡🫡
Thx, just tried upgrading it, the Intellisense speed is almost instant but can't deploy yet due to CI type check failing
https://github.com/drizzle-team/drizzle-orm/issues/988
GitHub
[BUG]:
drizzle-orm 0.28.0 type fails with drizzle-zod 0.5.0 · I...What version of drizzle-orm are you using? 0.28.0 What version of drizzle-kit are you using? 0.19.12 What version of drizzle-zod are you using? 0.5.0 Describe the Bug import { type InferModel, sql ...
cc @Dan Kochetov