Restoring 1.119.1 DB to 1.131.2 fails on Smart Search data
I'm trying to update my TrueNAS server from 24.04 to 24.10 and Immich along with it. My Immich db is not being migrated automatically by TrueNAS for some reason; Immich starts as if it's a brand new server. All of my images and users are gone on 24.10 with 1.131.2. So I took a postgres backup from the working kubernetes and am trying to restore it to the new docker container. I'm following the steps outlined here, just with a few modifications on the restore process:
https://github.com/immich-app/immich/discussions/10906
My restore begins working but fails trying to copy Smart Search data
My postgres database location in the TrueNAS Immich App config is the default ixVolumes. I stopped the immich containers other than pgvecto during restore and dropped and recreated the immich db so it's fresh.
Looking for any help on how I can fix the DB on 1.119.1 to get a working dump, or get TrueNAS to migrate immich correctly. I tried rerunning Smart Search job on 1.119.1 for All in the Immich Admin but this did not fix the issue after doing another dump. Thanks for any assistance!
19 Replies
:wave: Hey @candyman,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
2. :blue_square: read applicable release notes.
3. :blue_square: reviewed the FAQs for known issues.
4. :blue_square: reviewed Github for known issues.
5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.You should restore to 1.119 first of all
The restore should work regardless of which immich version I'm restoring for though, right? The pgvecto container on TrueNAS 24.04 and 24.10 are running PostgreSQL 15.12 so the dump should be compatible.
I'm following essentially this process recommended by Immich
https://immich.app/docs/administration/backup-and-restore/#manual-backup-and-restore
Are you sure it’s the identical version of vectors as well as pg 15.2 on both servers?
Yep, checked the image and ran
SELECT version(); on both containers. I'm stumpedHow did you check or confirm the vectors version
I checked the images used in kubernetes and docker - they're both using
pgvecto-rs:pg15-v0.2.0
so pgvector 0.2.0 it seemsCan you show the full command output of how you made the backup, any output, and the restore and full output
Theres tons of errors in there
You are not restoring into a fresh DB. As the logs say its critical that immich has not atarted up yet
you're also not following our docs, specifically
--dbname is missingThose are skippable errors though I believe, I just didn't clear the db out from my last restore attempt there. The COPY error is the real issue here. It fails at the same point when I drop and recrate the immich db and there aren't a ton of errors, see the attached output which is a fresh immich db. The immich server container is stopped while I'm working on the db container.
Sorry, I think I messed up that output; let me correct it
can you please share
line 25928 with context from the dump.sql.gzHere's lines 25927 through 25929 - nothing weird jumping out to me 🤔
@sogan Any thoughts?
can you post the output of the following from both old and new PG
can you also post the postgres logs?
The string representation for the data should be the same for any pgvecto.rs version so I kinda doubt it’s the issue. Are these backups generated by immich or by truenas? They used to do their own automatic db backups before any immich upgrade
If it was generated by truenas, then it could maybe be that the backup is done in a different way, like a different string encoding or something
I generated the backup from the TrueNAS shell (zsh I think?) using pg_dumpall over the kubectl exec shell. Tried with BASH just now, same result.
TrueNAS did leave backups in pgBackup (I think they're generated every time the App is updated) - image attached. The latest was quite old though so I thought I would generate a new one as recommended in the docs and by other people.
@Zeus
SOLVED: I was able to manually fix my database and migrate to Immich 1.131.3 on TrueNAS 24.10. Details here: https://forums.truenas.com/t/immich-app-postgres-database-does-not-migrate-from-24-04-to-24-10/38207/2?u=candyman
TrueNAS Community Forums
Immich App postgres Database does not migrate from 24.04 to 24.10
RESOLUTION I was finally able to fix this issue with a lot of trial and error. I manually migrated my immich database to Immich 1.131.3 on 24.10. Currently rebuilding smart search data and hoping everything works; it looks good so far! If anyone runs into this in the future as more people upgrade, good luck! I still have no idea why TrueNAS wa...