Questions for the team, once they actually get back, no rush: - Are there size limits for the cache?

Iitsmatteomanf Questions for the team, once they actually get back, no rush: - Are there size l...

I

Isaac McFadyen•9/4/23, 8:17 PM

AFAIK node_modules isn't cached.

I

Isaac McFadyen•9/4/23, 8:17 PM

Only ~/.npm is cached.

I

Isaac McFadyen•9/4/23, 8:17 PM

See James' comments above.

IIsaac McFadyen AFAIK node_modules isn't cached.

I

itsmatteomanfOP•9/4/23, 8:18 PM

I think it’s currently the opposite.

JJames How is the cache key determined, and how could I use caching successfully with m...

I

Isaac McFadyen•9/4/23, 8:18 PM

@itsmatteomanf

JJacob We're going to switch to caching npm/yarn global caches. Thanks!

I

Isaac McFadyen•9/4/23, 8:18 PM

Oh OK, yeah Jacob mentioned they were going to do it here but didn't say if that was current or not.

I

itsmatteomanfOP•9/4/23, 8:19 PM

Yeah, it’s planned for this week afaik, but we’ll see tomorrow

IIsaac McFadyen Only ~/.npm is cached.

J

Jacob•9/5/23, 5:48 PM

Working on making this change this week. Current beta still caches node_modules

L

Luke•9/5/23, 6:20 PM

Hello! I see only Next.js and Gatsby are supported so far. I have a few Vite projects and Nuxt projects I could test with once they're supported

JJacob Working on making this change this week. Current beta still caches node_modules

J

James•9/6/23, 11:02 AM

Are y’all planning on landing this before adding more folks to the beta? Excited to see this rolling out!

S

simpson•9/6/23, 7:16 PM

how do we tell if caching is enabled on our account?

Ssimpson how do we tell if caching is enabled on our account?

I

itsmatteomanfOP•9/6/23, 7:17 PM

There should be a message pretty soon after the build starts, in the logs, that is fetching the cache, and one at the end that is saving it (or skipping it for some reason)

JJames Are y’all planning on landing this before adding more folks to the beta? Excited...

J

Jacob•9/7/23, 3:37 PM

Adding more users shortly - expecting dependency cache changes by EOW!
Edit: Probably more like early next week. Possibly Astro support soon too

J

Jacob•9/8/23, 6:56 PM

Just pushed an update - dependency caching now caches global npm/yarn caches. Thanks again for the feedback @James

J

Jacob•9/8/23, 6:56 PM

Users who have submitted projects the last few days will be enabled in a bit

JJacob Just pushed an update - dependency caching now caches global npm/yarn caches. Th...

J

James•9/8/23, 7:22 PM

Amazing, thank you for the update and being so receptive to feedback! MeowHeartCloudflare

JJacob Just pushed an update - dependency caching now caches global npm/yarn caches. Th...

J

JohnDotAwesome•9/8/23, 7:49 PM

Do note that with the change in cache strategy, your previous caches will no longer match

J

Jacob•9/8/23, 9:27 PM

Astro support is now live party_blob

JJacob Astro support is now live <a:party_blob:818919990635266079>

I

itsmatteomanfOP•9/8/23, 9:47 PM

And... it works?

J

Jacob•9/8/23, 9:56 PM

should now!

I

itsmatteomanfOP•9/8/23, 9:56 PM

Let me retry!

I

itsmatteomanfOP•9/8/23, 9:57 PM

Looks like it does!

J

Jacob•9/8/23, 9:59 PM

All users that submitted projects now has build caching enabled for the submitted project. You will need to click

Enable build cache

Enable build cache

Enable build cache

Enable build cache in the

Builds and deployments

Builds and deployments

Builds and deployments

Builds and deployments settings.
For the first few users, the build cache toggle is currently ignored. But please enable it, as we will remove the override in the next week or two so that you can turn it on/off as desired.

NOTE: The "Enable build cache" options will show up in all projects, but it will ONLY have any affect on the project(s) you submitted.
Also: The

Clear cache

Clear cache

Clear cache

Clear cache button won't work until early next week.

Thanks!

S

simpson•9/9/23, 11:37 AM

Do we have to make any code changes on our side to reference the cache? I enabled it and it looks to work from what I see but it looks like my builds are the same or maybe a little slower

S

simpson•9/9/23, 11:39 AM

    
07:32:28.237    Success: Finished cloning repository files
07:32:29.372    Restoring from dependencies cache
07:32:29.401    Restoring from build_output cache
07:32:29.945    Successfully restored from dependencies cache
07:32:39.126    Successfully restored from build_output cache
07:32:39.664    Detected the following tools from environment: pnpm@8.6.6, nodejs@18.16.1
07:32:39.669    Installing project dependencies: pnpm install
07:32:41.489    Lockfile is up to date, resolution step is skipped
07:32:41.541    Progress: resolved 1, reused 0, downloaded 0, added 0
07:32:41.697    Packages: +615
07:32:41.698    ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
07:32:42.223    Packages are hard linked from the content-addressable store to the virtual store.
07:32:42.223      Content-addressable store is at: /opt/buildhome/.local/share/pnpm/store/v3
07:32:42.223      Virtual store is at:             node_modules/.pnpm

    
07:32:28.237    Success: Finished cloning repository files
07:32:29.372    Restoring from dependencies cache
07:32:29.401    Restoring from build_output cache
07:32:29.945    Successfully restored from dependencies cache
07:32:39.126    Successfully restored from build_output cache
07:32:39.664    Detected the following tools from environment: pnpm@8.6.6, nodejs@18.16.1
07:32:39.669    Installing project dependencies: pnpm install
07:32:41.489    Lockfile is up to date, resolution step is skipped
07:32:41.541    Progress: resolved 1, reused 0, downloaded 0, added 0
07:32:41.697    Packages: +615
07:32:41.698    ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
07:32:42.223    Packages are hard linked from the content-addressable store to the virtual store.
07:32:42.223      Content-addressable store is at: /opt/buildhome/.local/share/pnpm/store/v3
07:32:42.223      Virtual store is at:             node_modules/.pnpm

    
07:32:28.237    Success: Finished cloning repository files
07:32:29.372    Restoring from dependencies cache
07:32:29.401    Restoring from build_output cache
07:32:29.945    Successfully restored from dependencies cache
07:32:39.126    Successfully restored from build_output cache
07:32:39.664    Detected the following tools from environment: pnpm@8.6.6, nodejs@18.16.1
07:32:39.669    Installing project dependencies: pnpm install
07:32:41.489    Lockfile is up to date, resolution step is skipped
07:32:41.541    Progress: resolved 1, reused 0, downloaded 0, added 0
07:32:41.697    Packages: +615
07:32:41.698    ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
07:32:42.223    Packages are hard linked from the content-addressable store to the virtual store.
07:32:42.223      Content-addressable store is at: /opt/buildhome/.local/share/pnpm/store/v3
07:32:42.223      Virtual store is at:             node_modules/.pnpm

    
07:32:28.237    Success: Finished cloning repository files
07:32:29.372    Restoring from dependencies cache
07:32:29.401    Restoring from build_output cache
07:32:29.945    Successfully restored from dependencies cache
07:32:39.126    Successfully restored from build_output cache
07:32:39.664    Detected the following tools from environment: pnpm@8.6.6, nodejs@18.16.1
07:32:39.669    Installing project dependencies: pnpm install
07:32:41.489    Lockfile is up to date, resolution step is skipped
07:32:41.541    Progress: resolved 1, reused 0, downloaded 0, added 0
07:32:41.697    Packages: +615
07:32:41.698    ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
07:32:42.223    Packages are hard linked from the content-addressable store to the virtual store.
07:32:42.223      Content-addressable store is at: /opt/buildhome/.local/share/pnpm/store/v3
07:32:42.223      Virtual store is at:             node_modules/.pnpm

Ssimpson Do we have to make any code changes on our side to reference the cache? I enable...

I

itsmatteomanfOP•9/9/23, 12:46 PM

Nope, no code changes... the cache takes some time, so for very fast installs it won't change much, unfortunately.

S

simpson•9/9/23, 12:46 PM

No worries! Just wanted to make sure I wasn’t missing anything

Ssimpson Do we have to make any code changes on our side to reference the cache? I enable...

J

Jacob•9/11/23, 12:34 PM

Can you send me the deployment ID? I'd like to take a look - our goal is to make sure that at the very least, it doesn't slow down builds

JJacob Can you send me the deployment ID? I'd like to take a look - our goal is to make...

S

simpson•9/11/23, 6:47 PM

i'll try to grab it later, to be fair it was only like 10 seconds longer

J

josh•9/14/23, 11:01 PM

@Jacob any plans to support caching bundler/rubygems for Ruby builds in the future?

Jjosh <@85379843826413568> any plans to support caching bundler/rubygems for Ruby buil...

J

Jacob•9/14/23, 11:23 PM

Deff want to add support for more languages in the future, yes

How much time does your project take cached vs non-cached? (eg. when rebuilding locally)

JJacob Deff want to add support for more languages in the future, yes 🙂 How much time ...

J

josh•9/14/23, 11:40 PM

latest build is 2m 26s uncached at Cloudflare, 49s cached at Netlify

J

josh•9/14/23, 11:41 PM

this is a middleman site (quite similar to jekyll, should use the same ruby/node caching approach)

J

Jacob•9/15/23, 12:25 AM

Thanks!

I

itsmatteomanfOP•9/19/23, 12:12 PM

@Jacob did you think about the build cache not being invalidated on dependecy updates?

Iitsmatteomanf <@85379843826413568> did you think about the build cache not being invalidated o...

J

Jacob•9/20/23, 4:50 AM

A bit but nothing planned yet. This first iteration is intended to be very safe for most projects, at the cost of a few more cache misses. How often do you change your dependencies?

J

Jacob•9/20/23, 4:52 AM

What would you prefer as the cache key?

JJacob A bit but nothing planned yet. This first iteration is intended to be very safe ...

I

itsmatteomanfOP•9/20/23, 7:43 AM

It’s not the changing of dependencies (it happens often now as I am building the website, but will most likely lower), but the updates. Astro releases an update a week, add other packages and it’s very often.

JJacob What would you prefer as the cache key?

I

itsmatteomanfOP•9/20/23, 7:45 AM

Do we need a cache key for the output cache? I get dependencies, might make sense, although I am not sure what Netlify does, for example… but the cache output is handled by each package.

Iitsmatteomanf Do we need a cache key for the output cache? I get dependencies, might make sens...

J

Jacob•9/20/23, 1:52 PM

What if you have a different branch with vastly different settings? Maybe we could use the branch as a cache key instead? What if you change frameworks? (I guess we could rely on you hitting the Clear Cache button for that?)

J

Jacob•9/20/23, 1:56 PM

We were trying to avoid users having to hit Clear Cache and having things just work.

J

Jacob•9/20/23, 1:56 PM

But we weren't thinking about users updating deps every week

JJacob What if you have a different branch with vastly different settings? Maybe we cou...

I

itsmatteomanfOP•9/20/23, 3:17 PM

Oh, yeah. Branches for sure, I wasn’t gonna even mention that.

I

itsmatteomanfOP•9/20/23, 3:18 PM

Most frameworks tend to have their own cache folder, so it wouldn’t be a big issue…

Iitsmatteomanf Most frameworks tend to have their own cache folder, so it wouldn’t be a big iss...

J

Jacob•9/20/23, 3:24 PM

I wonder what would be a good way to surface that "hey you should probably clear your cache once in a while if you make big changes." Like I'd hate to be downloading/uploading 300MB of extra junk that's no longer used on every build

JJacob I wonder what would be a good way to surface that "hey you should probably clear...

I

itsmatteomanfOP•9/20/23, 3:25 PM

Maybe some sort of diff in the package.json? If it changes too much give a nudge? I don’t know…

J

Jacob•9/20/23, 3:27 PM

I want to add more config later - cache keys will probably be part of that. The current "play it super safe" approach will unfortunately come with some increased cache misses

J

Jacob•9/20/23, 3:27 PM

The hope is that it will still save users some build time if they build a lot. We can improve more edge cases in the future

JJacob The hope is that it will still save users some build time if they build a lot. W...

I

itsmatteomanfOP•9/20/23, 7:25 PM

J

Jacob•9/26/23, 8:40 PM

Heya! We released a few bug fixes & performance improvements. lmk if y'all see any issues!

Questions for the team, once they actually get back, no rush: - Are there size limits for the cache?

Similar Threads

Similar Threads

Similar Threads