Worker Deploys are Corrupted
18:02:39.394 Asset upload failed. Retrying...
18:02:39.394 APIError: A request to the Cloudflare API (/accounts/491ffb4e2aaea21bc19c73ec334e27c1/workers/assets/upload?base64=true) failed.
---
Just started happening now when trying to deploy a preview build.
What's interesting is that it then uploaded anyway, causing a corrupted build:
18:02:35.352 ✨ Read 1394 files from the assets directory /opt/buildhome/repo/.open-next/assets
18:02:37.029 🌀 Starting asset upload...
18:02:39.056 🌀 Found 3 new or modified static assets to upload. Proceeding with upload...
18:02:39.056 + /BUILD_ID
18:02:39.057 + /_next/static/bbHV35lxdDEEBhCISrqu6/_buildManifest.js
18:02:39.057 + /sw.js
18:02:39.394 Asset upload failed. Retrying...
18:02:39.394 APIError: A request to the Cloudflare API (/accounts/491ffb4e2aaea21bc19c73ec334e27c1/workers/assets/upload?base64=true) failed.
18:02:40.202 Uploaded 1 of 3 assets
18:02:40.460 Uploaded 2 of 3 assets
18:02:41.534 Uploaded 3 of 3 assets
18:02:41.534 ✨ Success! Uploaded 3 files (1268 already uploaded) (2.48 sec)
18:02:41.534
36 Replies
Hello! I see a connection drop from the client (wrangler). It looks like it was then re-uploaded. What's broken with your preview? Or did you see just an error and think maybe we didn't recover?
just an "Internal Server Error"
also, can't find anything in logs
Any ideas? Is it isolated to preview deployments?
Noticed an update that makes the "branch preview" more accurate, might be something that was introduced with this?
Are you able to share your preview url? If your worker is being invoked, then it's a problem with your user code. You can also check the observability tab in the UI to see if there's anything interesting there.
Can definitely confirm, this is a user exception, not in Cloudflare space
Very strange - no workers/infra-related changes; compiling & running works fine, just not within the worker
I recommend adding some console logs, try/catches, , and logging 🙂
Even running the
workers preview works fine (locally)...
I am pretty confident that it is related to cf workersYou would receive a well formed error (https://developers.cloudflare.com/workers/observability/errors/) if it were from workers control plane. Have you added a try/catch to try to debug yet?
The opennext app already has an ErrorBoundary
There is no other try-catch that can be ran - this is 100% a corrupted build
I guess I'll just wait it out lol
(even fully static pages are throwing the internal server error)
I believe it might be related to this^
Maybe the "already uploaded"/cached files are incorrect?
Do you know if it's possible to view logs of a preview deployment?
Nothing shows up for the preview deployment in the observability tabs - not even requests, etc.
Yeah you'd have to do a gradual rollout of 0% with this new version, then you can use a version override header to invoke
Checking for what I can see on our end!
Which worker name is this? I see 3 modified today
linkme-stg
Will have a look
Cloudflare Docs
Gradual deployments
Incrementally deploy code changes to your Workers with gradual deployments.
Well I'll be darned, may be an error in our assets flow. Checking..
Hope it is(n't)
(about to start losing my mind)
Ah yeah, so definitively-- it doesn't look like there are any matching assets on this path, we're falling back to invoking your user worker. The user worker (your worker) is returning a 500
Definitely recommend gradual rollouts (busted version at 0%) and then using the override header to force invocation of that worker
DEF a pain point of not having logs for previews... sorry about that.
alternatively if it's a staging worker, just fully deploy and you can roll back later 🙂
Thanks for your time!
Thank you
Yeah totally! Happy to help with gradual rollouts if you get snagged there!
Hey - sorry to ping you, but preview deployments are still f-ed
Deploying works fine, though
Same issue as https://discord.com/channels/595317990191398933/1440723603632033924/1440731013562106036
So from Will's looking, this was coming from your user worker
Have you looked at workers logs to see what the error is?
No, it isn’t
100% sure it’s an error on the preview deployments assets flow^
Assets are missing on preview deployments
https://discord.com/channels/595317990191398933/1440092870144168186/1440110929122693180
He said after checking it invoked your user Worker which is returning the 500
well ofc the worker will return a 500 error if it has no assets
the worker is an opennext web app (it expects the assets to be in place, and 0 assets are successfully uploading)
it only happens in preview deployments
(when running
wrangler upload)
the title is slightly misleading - probably something more accurate would be "Worker Preview Deploys are Corrupted"Define corrupted, I can't easily see assets but OpenNext is usually SSR'd - are you SSRing or using SSG?
corrupted meaning - it is missing a lot of JS that should be there
both, and both are f-ed
just an "Internal Server Error" & you can't even see any logs because preview deployments don't have any logs, and when deploying to the staging worker it works just fine
Ok that's not super helpful haha
d5bcc37c-linkme-stg.tech-491.workers.dev - it isn't finding a index.html (expected if SSRing) so invokes the user worker and 500s
ah yes.. previews not having logs is something i forgot :/exactly... assets are not uploading
only for previews btw
it used to work fine, and since ~yesterday they seem to be completely broken (no changes at all to the config, pretty confident it's an error on the CF side)
Looking at the prod one which will 200
This is also invoking the user worker
So both preview and prod invoke the user worker - meaning, there is no index.html asset and that is expected
yeah but prod has all the assets uploaded
So, yes, the issue is somewhere in the user Worker, I can't say why because well, there are no logs for either of us to see sadly
We aren't hitting assets though
I'm not sure where the feeling assets being missed is but we aren't hitting an asset in either one
basically, what I think is happening - the compiled server is looking for chunked JS assets, and they are not present
yes the compiled next server has a
dist folder with the minified JS
which it then imports dynamically
those are assets
I really don't think it is, as it works when you don't wrangler upload
and instead you do wrangler deployOk, we're using different meanings of assets.
I can't see if the code is trying to import a module that doesn't exist and since we both can't see logs, hard to tell too sadly
I can see that the uploaded build is missing the chunked JS from the dist
Is there an issue importing a module? very possible, not visibility i have but would at least still happen in the user worker
where are you looking to see?
going to send a SC
cc: @Dario on wrangler behaviour + opennext behaviour here
version upload/deploy use the same build path so should result in the same thing but also not sure if OpenNext is doing something special that's being missed