Cloudflare Developers•3y ago

I think John already knows what I'd ask for 😅

I think John already knows what I'd ask for

JJames I think John already knows what I'd ask for 😅

JohnDotAwesome•8/25/23, 5:11 PM

Bun?!

JJohnDotAwesome Bun?!

JamesOP•8/25/23, 5:11 PM

JohnDotAwesome•8/25/23, 5:11 PM

I think technically you could get it working... You're already setting SKIP_DEPENDENCY_INSTALL, right?

JamesOP•8/25/23, 5:12 PM

Negative, the only projects using it are the ones on your branch

JohnDotAwesome•8/25/23, 5:15 PM

I had someone else asking for bun as a package manager yesterday. That FR is up to 10 likes. I think if you ask @Vero enough times, she could make it happen :p

Vero•8/25/23, 5:15 PM

I've asked for that a lot too hahah

JamesOP•8/25/23, 5:15 PM

Vero•8/25/23, 5:16 PM

so yeah, keep asking and I'll keep insisting

JamesOP•8/25/23, 5:16 PM

@Vero bun pls

Vero•8/25/23, 5:16 PM

ahhahah, I'm actually sending my newsletter today and that's the request # one

VVero ahhahah, I'm actually sending my newsletter today and that's the request # one

DaniFoldi•8/25/23, 6:45 PM

Bun pls

itsmatteomanf•8/25/23, 7:24 PM

Filled the form

itsmatteomanf•8/25/23, 7:25 PM

Let's try one project, then go from there lul

itsmatteomanf•8/25/23, 7:28 PM

Ok, let's be daring and do two lul

James•8/25/23, 7:54 PM

With npmnpm, which directories are cached? ~/.npm~/.npm?

JJames With `npm`, which directories are cached? `~/.npm`?

JohnDotAwesome•8/25/23, 7:58 PM

Just

node_modules

node_modules

right now. What goes into ~/.npm~/.npm?

JJohnDotAwesome Just `node_modules` right now. What goes into `~/.npm`?

James•8/25/23, 7:58 PM

Oh.. that's a bad idea

James•8/25/23, 7:58 PM

Let me get you some docs, sec

JohnDotAwesome•8/25/23, 7:59 PM

Unless you're running npm installnpm install as a follow up to Pages running it for you, caching ~/.npm~/.npm wouldn't have an effect

James•8/25/23, 8:00 PM

The issue with caching

node_modules

node_modules

is that it can break across node versions, OS versions, and won't work with npm cinpm ci since npm cinpm ci always nukes

node_modules

node_modules

first. You want to cache the npm cache result, which is ~/.npm~/.npm unless you're overriding the npm cache location.

James•8/25/23, 8:00 PM

Github's own cache action hit this with early version, and you can see their docs about this here for example: https://github.com/actions/cache/blob/main/examples.md#node---npm

JJames The issue with caching `node_modules` is that it can break across node versions,...

JohnDotAwesome•8/25/23, 8:00 PM

We skip running npm cinpm ci altogether and (ought to) incorporate node version in the cache key

James•8/25/23, 8:01 PM

I thought the build image v2 ran npm cinpm ci and not npm installnpm install to give reproducible builds?

JJames I thought the build image v2 ran `npm ci` and not `npm install` to give reproduc...

JohnDotAwesome•8/25/23, 8:02 PM

With caching, under certain conditions, we know that step can be skipped altogether

James•8/25/23, 8:03 PM

I strongly suspect you're going to hit issues with caching the

node_modules

node_modules

folder. It's a path well trodden by other CI platforms and folks for years: https://stackoverflow.com/questions/42521884/should-i-have-travis-cache-node-modules-or-home-npm

James•8/25/23, 8:03 PM

In my project as an example, I have my site in a separate folder, so I cd site && npm ci && ...cd site && npm ci && ... as my build command

James•8/25/23, 8:04 PM

If ~/.npm~/.npm was cached, I would benefit, but with

node_modules

node_modules

cache I do not

JJames In my project as an example, I have my site in a separate folder, so I `cd site ...

itsmatteomanf•8/25/23, 8:05 PM

Why aren't you using Pages' root folder option?

Iitsmatteomanf Why aren't you using Pages' root folder option? 🤔

JohnDotAwesome•8/25/23, 8:06 PM

Shenanigans with monorepos and cwd's on Pages

JJohnDotAwesome Shenanigans with monorepos and cwd's on Pages

itsmatteomanf•8/25/23, 8:07 PM

I mean, mine is a monorepo, too...

James•8/25/23, 8:07 PM

yeah I probably could with this project, but root dir changing definitely causes (or caused?) weird issues in the past, so I just don't bother

JJames The issue with caching `node_modules` is that it can break across node versions,...

Jacob•8/25/23, 8:17 PM

Node version is a good callout, thanks!

James•8/25/23, 8:26 PM

How is the cache key determined, and how could I use caching successfully with my example project that does cd site && npm ci && ...cd site && npm ci && ...? If ~/.npm~/.npm was cached, worst case I have to download a couple new dependencies even if my package-lockpackage-lock, etc. changes

James•8/25/23, 8:27 PM

If I didn't have any nested-folder shenanigans, I'd assume updating

package-lock.json

package-lock.json

would invalidate the cache key and mean the entire

node_modules

node_modules

would be re-downloaded?

James•8/25/23, 8:28 PM

Caching ~/.npm~/.npm just seems like a better solution to me and also means that any post-install scripts still run. Let's say I have an npm package that doesn't change version between installs, but has a postinstallpostinstall that runs and pulls updated geoip data or something every time. That's not that uncommon of a use-case.

James•8/25/23, 8:29 PM

By caching

node_modules

node_modules

in its entirety and skipping the install step, that breaks that, vs caching ~/.npm~/.npm and just running npm cinpm ci again which'll use that cache, run any normal hooks/scripts, and everything works as expected. You get the benefit of the cache, and any kinda of postinstall or other weird hooks/scripts run as expected.

James•8/25/23, 8:29 PM

Skipping the npm installnpm install step entirely seems like a bad idea

itsmatteomanf•8/25/23, 8:31 PM

Uhm, was my project enabled?

22:30:34.197    Uploading to dependency cache
22:30:34.254    Skipping build output cache as it's not supported for your project

22:30:34.197    Uploading to dependency cache
22:30:34.254    Skipping build output cache as it's not supported for your project

JJames Skipping the `npm install` step entirely seems like a bad idea 🤔

James•8/25/23, 8:31 PM

I would really think heavily about this @Jacob @JohnDotAwesome . My DMs are open for further discussion, but I really encourage you rethink the design with

node_modules

node_modules

here

JJames I would really think heavily about this <@85379843826413568> <@78687712646795264...

Jacob•8/25/23, 8:36 PM

We're discussing internally, thank you for the feedback!

Iitsmatteomanf Uhm, was my project enabled? ``` 22:30:34.197 Uploading to dependency cache 2...

Jacob•8/25/23, 8:37 PM

It's caching dependencies, but not "build output" such as .cache / public for Gatsby. The log messages are pretty rough right now - we will clean them up soon

JJacob It's caching dependencies, but not "build output" such as .cache / public for Ga...

itsmatteomanf•8/25/23, 8:37 PM

Oh, gotcha

It's currently saving me 8s, so I went from 83s to 75s...

itsmatteomanf•8/25/23, 8:37 PM

Of course there is variations between builds

JJacob We're discussing internally, thank you for the feedback!

James•8/25/23, 8:38 PM

Thanks, please just let me know if I can provide any more info or feedback! The biggest breakage potential here is any kind of postinstallpostinstall, etc. scripts, outside of just unnecessary re-downloads. Skipping the npm installnpm install (etc.) step is gonna break any workflow like that that relies on something running, and it's not uncommon for npmnpm packages to do that kind of thing.

Iitsmatteomanf Oh, gotcha 🙂 It's currently saving me 8s, so I went from 83s to 75s...

itsmatteomanf•8/25/23, 8:39 PM

The actual build is ~6s.

Jacob•8/25/23, 8:43 PM

We have some optimizations left to be made, so hopefully that will improve!

Jacob•8/25/23, 9:29 PM

something I somehow didn't realize before this project is that a seemingly simple project has like 500-700MB of deps

James•8/25/23, 9:31 PM

Yeahhhh

node_modules

node_modules

bloat is fun. So many libraries end up publishing both build and source files, tests, docs, etc. inflating file size, and optimising output just isn't a priority

James•8/25/23, 9:32 PM

Simply installing

wrangler

wrangler

and nothing else in a project for example gives me almost 100MB of

node_modules

node_modules

I think John already knows what I'd ask for 😅

Similar Threads

Similar Threads

Similar Threads