System crashed and corrupted multiple files
Hello all, so I made an upload last week and it somehow crashed immich_server (even brought down my entire docker daemon, which is exceedingly strange).
After restarting everything, quite a few images were corrupted, and all the new uploads ever since also becomes corrupted.
I've tried regenerating thumbnails, re-pulling and redeploying, but nothing works. I am planning on re-creating everything from scratch, however, I am hoping to find a solution to this, in case it happens again in the future.
some error logs (couldn't find the logs when immich_server crashed):
49 Replies
:wave: Hey @Mai,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:Could you add your compose/env and host environment @Mai ?
Have you ensured that your hardware isn't failing?
the compose:
env variables:
That was my suspicion at first, I've thoroughly checked everything and made sure it is fine.
Seems like a Synology NAS hosting it
Which model do you have?
(Right now my mind is going towards too little memory)
It is on Synology NAS DS920+, I am deploying immich using Portainer. I've upgrade it to 20G of memory, even with everything is on, it rarely go past 30%
Yeah 20 should be fine for the heaviest users, NASes often only have 4G which is just enough until you start uploading things
If you upload new images, do they appear in the GUI?
Even with an "Error"
or just nothing
yep it apprears but with "Error"
most of the old pictures were still intact, but all the new uploads are broken.
Ok but the new ones do upload, that means the thumbnail generation is broken π€
the thumbnail is definitely broken, and I've tried to click on it and the "enlarged"(I am guessing it will show the original picture) pic is also broken.
No the enlarged one shows a larger thumbnail π
if you download then it will use the original
thumbnail is maybe a bit misleading because the large ones are 1440p
ah I see, so yeah, the thumbnails are broken
haha that is not really a thumbnail
yeah sorry the GUI calls them "preview" but they are stored in the thumbnail section nonetheless
You don't store the thumbnails seperately so I'm really scratching my head here
Do you know which upload caused this?
yep, I've tried deleting it and re-upload again, but it is still broken
but I have teared down everything so I am unable to produce any logs
Partition isn't full or anything ?
There are permanent database-specific logs in DB_DATA_LOCATION/log(s) (I always forget whether there is an s)
oh that's a good point, I've recently become addicted to Private Tracker/BT download, took up a lot of space, but it never went down to less than 100GB
ah, i will write that down for future reference
Yeah unless your file happened to be 100GB I don't think that will have caused it then π
yep, so...any idea about what should I do to prevent it from happening again?
Is the file anything you're willing to share ?
kinda nervous, I love immich so much, it is the coolest app in my homelab
sure
If you want to, zip it up and upload it here or in private
but I have deleted the old immich folder though
don't straight upload it to discord because it will clean it up
let me see if I can dug up from Synology's recycle
if I can find it, what files should I upload?
Oh I was wondering if you knew the specific file it crashed on, if not then maybe it will be a wild goose chase π
haha, it is just a few pictures took on an iPhone, I don't think it is any different than the other pictures
it is very strange, becuase the said upload crashed my entire docker daemon, I thought dockers are "containerized/compartmentrized"
still got no idea what happened
oh btw, while I am rebuilding, I've run into new problems...wondering if you can also help me with it?
Sure, just tell us π
Which iPhone?
thanks! so
it is an iPhone 16 pro
the small one, not pro max
and it was uploading using the immich app
so I am trying to rebuild today, and I wanted to bind external libraries, as you can probably see on my compose file:
/volume1/homes/lakrymosa/Photos/PhotoLibrary/M/Mina:/volume1/homes/lakrymosa/Photos/PhotoLibrary/M/Mina. I had this particular binding running fine in my old immich, but today with the new build it just won't workJust curious because I recently got an iPhone 16 and I'm seeing weird thumbnails because the live photos have multiple previews embedded in them
very strange, I couldn't remember how I managed to bind it on my old immich, I binded it on the first day I deployed immich
hmm it make sense, could this be the reason?
but live photos isn't new though, I probably uploaded live photos before
I don't think it should be this, just curious if there could be some connection
Pretty sure this should work fine :/
Unless there is some permission you still need to give the container on the Synology side
yep that's what I thought, and the link is validated successfully, but the picture won't show up, and the library says 0 photos
yep I also believe this is the case
I am not very code-savvy, so I've tried some easy one such as including
user: 1026:100 in the compose, or putting PUID PGID in the environment variables, but still won't work
and crashed it again, had to rebuild multiple time, still stuck
the user: 1026:100 is my Synology administrator ID
I think I might need to SSH into it and try those "chmod+numbers" thing
Mraedis is Gandoff's elv name right?
Gandoff the white wizard
oh and btw, I am also reading the "data dump" section of the immich doc, it says I can use it as a backup mechanism? not sure how to do it thought...I am worried that immich might crash and files become corrupted again, and I have to rebuild...uploading a ton of photos and waiting on the system to recongize all the faces is a nightmare.
is there an easy way to backup? I am wondering, can I just copy and backup the "immich" folder regularly, or use those fancy "snapshot" functions, if things go south, I can just replace the immich folder with the backup and everything will be back online?
and another stupid question: there are two other family members using my immich, but I have some private photo that I rather keep it to myself, is there a way to do it in immich? I am the adminNo, Mraedis is my personal user name π
the backup folder is where the database dumps are stored
these are database only and are done once every 24 hours unless you've changed the settings
database dumps are very easy IMO
photos are per-user, so if you share the user then no there is no way, but if you have separate users then they can't see anything unless you turn on partner sharing
the admin also can't see anything unless they navigate to the files on the host
I am not quite following, so do you mean that I just need to backup the datebase bump folder?
this part is crystal clear π
https://my.immich.app/admin/system-settings?isOpen=backup
You can also adjust the frequency
And yes, having a copy of the 'backup' folder should be basically it as far as the database is concerned
thanks MraedisοΌwill check it out!
@Mraedis Hello Mraedis, I can really use some help here! I build a fresh one using the compose and env. from immich official guide, but all the uploads are broken. I rebuilt immich multiple times yesterday trying to fix the eternal library mount, everytime I rebuilt, I always uploaded a few photos to make sure immich is functioning properly, and it always worked well.
So right now I am scratching my head here, I haven't done anything differently. I've also tried force refresh/using a different brower, tear down everything and make sure it is a fresh rebuild, but it is still broken.
should I upload my compose and eni variables here? but everything is just simply copy and paste from official docs
Did you also add external libraries?
Thanks for replying so quickly! nope, I didn't mount external library
What's in the docker logs?
For immich_server, all the logs are green, I don't see any errors:
The redis looks fine too:
Postgres also looks fine:
machine learning:
All the logs above are from a fresh rebuild (built just 5 mins before I post on Discord), so all the logs are from the very begining
Really have no idea here. Maybe I rebuilt way too many time yesterday, and somehow the docker daemon cached some werid errors and corruptted my new build? doesn't make any sense. Should restart my host (synology NAS)?
Not seeing any errors in your logs
What do you mean by "uploads are broken" btw
the thumbnails are broken
could you go into the admin section and press "missing" on the thumbnails job
report back how many it is queueing
in the admin section, "GENERATE THUMBNAILS" tab, it was showing 0 active, 3 waiting. After I pressed the "missing", 4 is waiting
this is weird, I just checked my compose file again, I did not include any external bind mount, but here it is showing an external library waiting
there is nothing in the external library section, and I didn't add any.
curious indeed
Did you check the external library section in the admin panel to make sure there is none?
But it sounds like your redis cache is a bit broken
yep, there is none
might have to reboot my synology nas, will report back
@Mraedis Hello Mraedis, I am happy to report that after restarting my Synology, immich is working fine now (I didn't rebuild after restart, it is the same build as before restart).
hey Mraedis, here are some additional updates: the bind mounts are also working now, at first the bind mount pictures are "broken", however, after a container restart, thumbnails are generated properly.
Thanks a lot for your help, really appreciate it!