PNPM cache not working on Workers build
Have not been able to make caching work for pnpm in workers build. Couple of issues I found:
Confusing docs:
- looking at the docs it states that the caching folder for pnpm is
- Blasted the build cache - Run a couple of builds to populate cache. I'd expect the output of CI to be something like: -
.pnpm-store
- When outputting the location using pnpm store path it says it is /opt/buildhome/.local/share/pnpm/store/v3
Forcing a location still downloads files
Tried debugging it by:
- Setting SKIP_DEPENDENCY_INSTALL true (I can see it working)
- Set manually a cache location pnpm config set store-dir ~/.pnpm-store (Validated it is set with an echo before and after)
- Set script to manually install Manually run pnpm install --prefer-offline- Blasted the build cache - Run a couple of builds to populate cache. I'd expect the output of CI to be something like: -
Progress: resolved 2476, reused 2459, downloaded 0, added 2476, done
(Notice the downloaded 0)
- Still seeing:
- Even when running this a couple of time with the same build-command
Not sure what heuristic is used for caching (I imagine pnpm-lock.yaml ? Would be great to know that 🙏 )
No build system options on workers setting
- On the docs, I see a section for v2 to v3 Migration (here)
- Thought this might be the case. But could not find that option anywhere on my workers.
----
FWIW ...I'm using this in a mono-repo setup with a pnpm-workspace.yaml file28 Replies
Following this -- we are also using a pnpm workspace for our Workers and having the same issue (I see a
Failed to restore dependencies from build cache. Skipping. at the start of each build, even though it says it saves them at the end of each buildI think package dependencies and build dependencies are treated differently from what I understand.
JIC you are using something like... remix/react-router
It does not provide a cacheable build output as opposed to other frameworks
Oh, yeah you're right, so I guess that message is unrelated. Still seeing the same behavior as you with "reused 0" on all the dependencies (on top of it not filtering so it installs everything 🙃)
I've raised this internally
Hi @fforres 👋!
Would it be possible for you to provide your build uuid? It'll be in your address bar like so
.../production/builds/<build-uuid>@Maximo we should update the docs. We use the default store dir on linux which is
~/.local/share/pnpm/storeFor sure, I have a couple of em.
@Maximo
28c914ff-be0d-4f77-bb59-4c94ec9118bf
This one is skipping dependency install.
(And Setting the store-dir to ~/.pnpm-store)
Question @JohnDotAwesome . Does this only cache on the default pnpm store dir? Or if I change it, it uses the updated folder?)
(And if so, I'd be very curious on the heursitics... does it generate a cache per branch? Does it have a series of "nested" keys like github's action?)
If it helps, I just run a test with default store-dir.
Made a couple of CI runs with this command
This is a run with it db8ee2d9-2507-4e24-bf54-af0a5845acb7
(With SKIP_DEPENDENCY_INSTALL true)
On both ls, i'm able to see the files there 🙂
But the progress still shows downloaded fields on subsequent runs
We only detect specific directories to cache. So changing it to a custom directory will not work
Gotcha, 🙂
I'm going to have to continue looking into this tomorrow. I apologize for the issues here!
Appreciate it @JohnDotAwesome 🙏
I've got to be missing something. I fixed pnpm caching last week. I have a personal project that uses pnpm workspaces and caching is working fine on that repo. Although, pnpm install is so damn fast, caching hardly makes a difference anyway
Oh actually there it is... the extract cache is failing for you. Interesting...
@fforres just double checking - have you tried clearing the cache for your builds yet?
Yes I have, will try again on new run.
Is the cache scoped per-branch/branch/etc?
I assume it hashes the package files, but aside from that, wondering if there's anything else to consider
@JohnDotAwesome ? 🤔
Cache will effectively be scoped per worker. The key is computed from your project's lockfile, so it can be shared between branches
Something about your cache store is breaking our extraction process. I'm trying to reproduce it on my own repo now
Did a new run 🙂
- Cleared cache
- First run here:
8f951107-26a3-438a-8449-ca22946d3198 (Downloaded packages. Saw the Uploading to dependency cache at the end)
- Re-run here 3144e5fb-3639-49f0-a6f2-7cdd2739a631 says (Failed to restore dependencies from build cache. Skipping)
(Org ID: b75ee9798c9c48a874af240b29724270 in case it helps)
got a mono-repo with a few sub projects (11)
Aside from that ... i don't think we have something custom.
Could it be that we did a few changes to the .npmrc file?
I made some updates to speedup codex a bit ago 🤔
I don't think it should affect, bu JICUnlikely... But we'll see 🙂
hey @JohnDotAwesome !
RE: ping on this 🙏 Is there any way I can help?
Happy to jump on a call/give you acess to our repo, or anything
Also, I know you mentioned:
Happy to help in any way
I have a personal project that uses pnpm workspaces and caching is working fine on that repo.Is that public? Would love to compare setups maybe I'm missing something 🙏
Happy to help in any way
Hey @fforres apologies I had to shift focus. Jumping on a call would be great. I’ll DM you
👋 not sure if you've been able to see this? Still seeing issues 😦
👋 @JohnDotAwesome ? 🙏
Or @Vero 🐙 maybe? 🙏
In case it helps:
Our organization id is:
b75ee9798c9c48a874af240b29724270
and a recent run build-id that did not picked up a cache: 0adf3706-5b8f-49fb-a004-65f8cdfd8bd5
Some more context on this.
Another thing on why we are looking into have pnpm cache working, is that we usually see timeouts on the pnpm registry for some package installations, that made our builds to increase on 30+ seconds, or more for example when they fail more than once.
We seem to also be getting hit by this, i.e., we see "Failed to restore dependencies from build cache. Skipping." on each build
We also experience the same behaviour, pnpm monorepo, build command
pnpm install --frozen-lockfile
Org id: 814e8d36e1399315694670071a13c144
Build id: 5146c938-7a2e-45d5-9ac0-3930b9e1eabf
👋 hey folks! I've escalated to the Builds team for them to look into this again. Thanks for letting us know!
Hey folks! I took a look at this today and got a fix ready and will ship monday. Confirmed cache restore worked on a pnpm monorepo with 3k deps. Thank you for your patience 🙏
awesome! I guess it's still yet to be deployed as I see the same behaviour. Do give us a shout when it goes live!
Yep sorry about that - getting it shipped today for sure and will let you know!
this went out yesterday afternoon btw! Lmk if you still see any issues
awesome! Interestingly, hasn't made builds any faster.. I guess the download speeds are fast enough for it not to really matter.. (although it does avoid the occasional npm registry connection timeout)
Yeah, pnpm documents that caching may not actually speed things up unfortunately https://pnpm.io/continuous-integration
re: timeouts - we have some changes releasing soon that should make certain network requests more reliable, so hopefully we will see fewer issues with npm like that