Immich_Server stopped unexpectedly yesterday, and now assets are not loaded on web and iOS app
Hi, I updated to v1.131.3 yesterday afternoon, and then subsequently tried to upload some 150GB worth of new assets (around 6000 assets in total). Within about 15 minutes of this, the server container stopped, and then restarted. I thought it all went well, but since found out this morning that only about 75 new assets were uploaded, everything else was skipped due to "TypeError: fetch failed". On trying to use my app this morning, I also see that the assets are not loaded and will not play any videos. App screenshot below. the server log also shows some errors, but I'm not able to decode what's happening and why nothing seems to be working. Any advise please?
20 Replies
:wave: Hey @Vat.Kris,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.Server logs for the relevant time (3:30 PM on 8th Apr) and app screenshot on here
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:
Out of curiosity what are your system specs? All 150gb coming from a mobile device?
No, locally from a Synology NAS.
Immich is also hosted on the NAS (docker)
What's the specs? You may have overloaded the system
It's a DS220+, 18GB RAM, 2 core CPU 2GHz
And I have uploaded 150GB before with no issues just FYI
+ will that affect already existing assets being shown on my app and web timeline? Videos do not play on the browser either
That's a pretty anemic cpu for a mass upload like that. I have no doubt it may have crashed your container.
Videos not playing in browser may be from them not being transcoded becuase of the crash. Not sure about existing assets now having problems on the web.
Can you look at some of those id's from the error log and see if they are corrupted photos?
The log from the failed upload or from the server?
I mean - the assets which failed to upload or the already uploaded assets which are not loading on the app/web?
OK so I checked both - assets that failed to upload and already uploaded assets - all of them are ok, and not corrupted. I'm able to access / view / download them etc.
Sorry, when I say "all" I mean the sample assets I checked lol! But I'm pretty sure they're all ok.
The server just crashed again.
I tried running some transcoding jobs, and the server crashed. Could it be something in transcoding that's causing this?
New logs from server
No other jobs running, no new uploads. Nothing else at the moment that could have caused this as far as I'm aware.
It’s probably just getting overloaded because the system is so weak
Transcoding puts a lot of stress on it
So my solution would be to turn off transcoding?
Sure but a lot of videos won’t play
I'm confused because I have 9600 videos on there, many of which had to be transcoded originally. Nothing has changed, the original transcoded videos are also on there. Running it on a few missing files shouldn't break the system no?
Right, and I don't want that either!
The logs are not readable can you upload a text file please, not html
Any luck?
Hi team, any advise for me please?