Pi5 crashes a couple of minutes after Immich starts - after upgrade of Immich - all thumbnails lost
I got a pi5 8gb running off a 1TB nvme, with a secondary 2TB nvme (X1004 geekworm 2 nvme adapter) for more storage running raspibian converted into an Open Media Vault setup.
its been running with no problems until I upgraded immich to the newest edition. That caused all my thumbnails to stop working.
A lot of the images also shows up as 0byte size if I try to download them.
Running the thumbnail generator turns the entire Pi off.
as long as the thumbnail job isn't running, the pi does not crash.
29 Replies
:wave: Hey @Chad_Maverick,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
2. :blue_square: read applicable release notes.
3. :blue_square: reviewed the FAQs for known issues.
4. :blue_square: reviewed Github for known issues.
5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.Sounds like something isn't mounted to the container
You mount points might not get mounted correctly
Or worse, it was not properly mounted at all from the start 👀
sudo docker ps -a | grep immich
a145fab60891 ghcr.io/immich-app/immich-server:release "tini -- /bin/bash s…" 23 hours ago Up 22 minutes (healthy) 0.0.0.0:2283->2283/tcp, [::]:2283->2283/tcp immich_server
57e75d84e7de ghcr.io/immich-app/immich-machine-learning:release "tini -- python -m i…" 23 hours ago Up 22 minutes (healthy) immich_machine_learning
1fff0e478d1e tensorchord/pgvecto-rs:pg14-v0.2.0 "docker-entrypoint.s…" 4 months ago Up 22 minutes (healthy) 5432/tcp immich_postgres
6f581146c443 redis:6.2-alpine "docker-entrypoint.s…" 4 months ago Up 22 minutes (healthy) 6379/tcp immich_redisI mean the mount of your external drives
lemme check.
lsblk, right?
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 38.7M 1 loop /snap/snapd/23772
loop1 7:1 0 44.3M 1 loop /snap/snapd/24509
nvme0n1 259:0 0 1.9T 0 disk
└─nvme0n1p1 259:1 0 1.9T 0 part /srv/dev-disk-by-uuid-40a79b2c-8b24-425e-ac71-c77c70747371
nvme1n1 259:2 0 931.5G 0 disk
├─nvme1n1p1 259:3 0 512M 0 part /boot/firmware
└─nvme1n1p2 259:4 0 931G 0 part /var/folder2ram/var/cache/samba
/var/folder2ram/var/lib/monit
/var/folder2ram/var/lib/rrdcached
/var/folder2ram/var/spool
/var/folder2ram/var/lib/openmediavault/rrd
/var/folder2ram/var/tmp
/var/folder2ram/var/log
/COuld you
sudo snap listchad@Pi5:~/immich-app $ sudo snap list
Name Version Rev Tracking Publisher Notes
snapd 2.68.4 24509 latest/stable canonical✓ snapdFor the pi, sometimes Docker could start before the drive get mounted
Yeah but.. if it has been running fine it might have been the other way around. Using the image volume to store things 👀
are the pictures on your drive as expected?
Most pictures from this month have "error loading image"
it seems like pictures from April and earlier are mostly ok?
it does not look like I specified any fancy directories for where Immich would store itself nor the database.
All of it should be on nvme1n1p2 partition
DF -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 796M 3.8M 792M 1% /run
/dev/nvme1n1p2 918G 322G 550G 37% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 5.0M 12K 5.0M 1% /run/lock
tmpfs 3.9G 0 3.9G 0% /tmp
/dev/loop0 39M 39M 0 100% /snap/snapd/23772
/dev/loop1 45M 45M 0 100% /snap/snapd/24509
/dev/nvme1n1p1 510M 72M 439M 14% /boot/firmware
folder2ram 3.9G 425M 3.5G 11% /var/log
/dev/nvme0n1p1 1.9T 126G 1.8T 7% /srv/dev-disk-by-uuid-40a79b2c-8b24-425e-ac71-c77c70747371
folder2ram 3.9G 0 3.9G 0% /var/tmp
folder2ram 3.9G 664K 3.9G 1% /var/lib/openmediavault/rrd
folder2ram 3.9G 3.8M 3.9G 1% /var/spool
folder2ram 3.9G 13M 3.9G 1% /var/lib/rrdcached
folder2ram 3.9G 4.0K 3.9G 1% /var/lib/monit
folder2ram 3.9G 16K 3.9G 1% /var/cache/samba
tmpfs 796M 4.0K 796M 1% /run/user/1000
Most pictures from May simply do not work, while 1 out of 3 dog photos the 16th of may (one at 11:37 gmt+2) are working, the other pics in May do not work.
I'm thinking I'l try to delete all the may pictures, and reupload them, and see if that changes anything.
I deleted all the images from May.
most of the images did not get reuploaded when I chose sync form my phone
(I think Immich might remember the hashses or some way of seeing that these images were deleted and should not be reuploaded?)
Also restarted the thumbnail checker job, and it still crashed.The entire Pi crashes, right?
What is your job concurrency set at
Yes
before I updated to the newst version, I had it above the default.
couple of hours ago, even when I had all of them to 1, it would still crash.
Now this newest test, I pressed the "reset to default" and saved before I tried agaon
Current (default) settings
Generate Thumbnails concurrency 3
Extract metadata concurrency 5
External Libraries concurrency 5
Sidecar metadata concurrency 5
Smart Search concurrency 2
Face detection concurrency 2
Facial Recognition concurrency 1
Transcode videos concurrency 1
Storage template migration concurrency 1
Migration concurrency 5Yeah just wondering whether your Pi is getting enough power/cooling 😛
Try setting a limit like:
For the immich_server container, the indent level is the same as image/ports/volumes..
I'm wondering the same thing.
the X1004 geekworm adapter has been dealing well with single nvme for a long time, and dual nvme drives ok ish for a good while too.
I've changed out my RPi5 PSU personally because it kept flaking
Chinesium 🥲
you know what??!
It seems like the current USB-C power thing I'm using is a knock off that I replaced a while ago!
I'm gonna dig through my stuff (moved twice) to see if I still have the same adapter that I ordered to be compatible with the nvme drive somewhere!
yeah Power supply is very important to keep up with the CPU load
if it cannot reach the constant current that the CPU needs during peak load, it will crash the system
I'm making my poor machine run like crazy right now, making it do a lot of jobs, a lot more than before.
And I can hear it working, and showing no signs of shutting down yet!
With the new PSU?
I garbage any cheap USB-C PSU I get
It's just not worth the misery
Yes!
I went from the no? brand power adapter to this "Geekworm Raspberry Pi 5 Power Supply 27W 5V 5A USB C, PD 27W Power Adapter" by the same makers of the dual nvme adapter I use.
Yeah GeekWorm is pretty great
This is the name of the adapter that was causing the issues.
ARyee 45W USB C Power Supply Charger Laptop Type C Charging Cable for Lenovo ThinkPad, Chromebook, Yoga; Dell XPS; HP Chromebook, X360; Huawei Matebook Laptop Type C Charger ""45"" W
🤣 ðŸ˜