Images not showing thumbnails or metadata
It looks like recently some images and videos uploaded are not showing any thumbnails or metadata. A refresh of the thumbnails and metadata doesn't seem to resolve the issue. If I download one of the images, I can view it normally and can see using the exiftool CLI that there is full metadata on it.
71 Replies
:wave: Hey @dekzter,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.Latest version. 1.31.3, no recent changes to server, deployment, infra, etc.
Plese post your compose and env. Also go to one of the photos, run the thumbnail job and grab the server log after
compose: https://pastebin.com/Z4PqKNbv
version: "3.8"Are you using docker-compose or docker compose?
docker compose
No logs after thumbnail jobs initiated. But I do see this line frequently in the logs:
ERROR: could not open file "base/5/2687": No such file or directoryvolumes: - ${NAS}/docker/immich/postgres:/var/lib/postgresql/dataDo the containers run on the NAS or is the NAS remote?
Run on the NAS
storage is all local - pooled drives with mergerfs
Looks like your database is corrupted
I would not ever use mergerfs for Postgres
I've been running immich like this for maybe 2 years with no issues yet.
But yeah I can move the DB
You’ll probably need to restore from backups
How can you tell there's db corruption?
And if I restore from backup, what happens to the files that exist in the already uploaded directories from my devices?
They will show up as unmanaged by immich in the (hidden) repair page and you can re-upload
This means there are files missing from the Postgres database
Unmanaged, it can't re-import from the existing location? I'm afraid that some may have been removed from the devices already
Maybe I’m confused. How old is your backup?
Indeed it can't, but you can download them
The default settings are daily database dumps, you don't have those?
download as in... copy from the upload/library folder
Well this is concerning - I've got backups configured in settings, but my last local backup is from November in the folder..
That’s probably when your database has been corrupted since
oh boy
would I see any logs for a failed backup?
hmm, doesn't look like any logs from midnight til 3am today. Backups are scheduled for 2am
There should be. I’ve seen it when testing
I've got nothing the past 24 hours except when an upload attempted a duplicate up, it errored from what I assume is duplicate checksum, and then some websocket connect / disconnect messages
At least for the server container
It appears the first failed image thumbnail was April 10th
So what's the recommendation? Restore from latest backup and reupload all images since then?
Try and see if you can get a database dump going manually, see what the error, if any, is
just a pgdump?
yeah
Well that's not good:
:/
All my backups are .gz.tmp files, is that right?
I gunzipped it and it looks like it loaded in just fine, just wasn't sure if .tmp was correct
gz.tmp? that does not sound right
Didn't think so..
If I restore from backup, how can I confirm the backup was successful / not corrupt?
how large are the files?
468M
A restore via psql worked, and a re-dump of that completed without error
should be fine then. must be an old renaming bug fixed since then
We are sorry this happened. We will be clarifying this in the docs. #17676
Basically you will need to re upload images since November , either from device or from the immich server
I can upload via CLI I think right? Should I reupload via cli directly from the data/library directory and it'll ingest / rename properly or should I remove / move any files first and then upload
I would do the restore, then I would probably visit the
/admin/repair page to get a list of Untracked assets, move them OUT of the Immich folder, then upload using CLIGot it. Doing that now.
I've also moved my DB off of the mergerfs mount
oof, seeing a ton of errors now
oh yes, hopefully you did the restore onto a non merger mount? from the beginning
ohhh you will have to roll back the server .. ofc
(I think)
It won't run migrations since the backup?
actually we haven't had a breakig change since Nov .. so it should work
are those errors from the fresh restore or from the old corrupt db?
fresh restore
I just started the containers
can you connect to the database now and see if that column does exist? it should do the migrations, but this is a pretty big jump which is hard for us to test..
perhaps it just needs a restart of the stack as there may be a race condition
Let me check
No it's not there
ok, you might have to walk through some of the versions then
@sogan any thoughts on these migrations failing?
can you run
select * from version_history; in the restored DB?latest ist 1.120.1
I am looking at my history, I skipped 125, 127, and 130. so if you need to walk it version by version at least I can confirm those shold work
Ok, I'll start back with 120.1 and go from there.
you can skip all 0.0.x (patch) versions
and yes let's see if we can get 120.1 at least running
Welp - when I started the container, it looks like it triggered a DB backup and wiped all other backup files, just FYI.
Not sure if it deletes based on age, but should prob. keep at least X number of recent backups
Well that's not good
I think the real question is why the tmp files even persisted
So should I even bother or just start fresh?
I wonder if the initial dumps were all failing as well
The corruption may go further back
Right...
Well I guess I've got my weekend planned out now.
#13995
[Pull Request] fix(server): attempt to delete failed backups immediately after failure (immich-app/immich#13995)
that fits the timeline here, unfortunately. those earlier dumps are probably incomplete
if you need to do a full re import you can use the CLI and just point it at the old library
Yeah, going to do that.
Less concerned about that and more about the time to regenerate all thumbnails / video encodings / all machine learning 😭
Sorry about that. I would say investigate the ML models now so you can start with the one you want, if you don't want to use the default one (not very good)
also if you use our new docker compose there are some health checks we have added to the DB that may have caught this earlier
Oh yeah, I never did anything with that.
Any suggestions?
but I'm certain it was realted to mergerfs
https://immich.app/docs/features/searching/
Can I disable ML to begin with and turn it on after I research models?
sure, just put the job on pause after it's triggered by any image
Yeah, I've still got a ton of photos with this new install with no thumbnails and not showing the image when they're opened
Those are the thumbnail generation jobs that need to run