Migrating from docker → proxmox LXC, restore failing - better way?
I recently moved my homelab into proxmox, which now has zfs via truenas, and my immich service moved from a separate device into an immich LXC (here). I started with process in
Immich v1.135.3, and both systems are currently locked at that version.
I have a full DB backup as described in the backup/restore docs.
The proxmox/immich setup has a different user/role setup, so I
- chopped the first ~40 lines where users/roles are dumped,
- updated all 'OWNER TO' immich except for the postgres and template1 DBs (matching proxmox virgin setup),
- and attempted to restore with a command like this:
cat ./immich-restore.sql | sed "s/SELECT pg_catalog.set_config('search_path', '', false);/SELECT pg_catalog.set_config('search_path', 'public, pg_catalog', true);/g" | ./psql-immich
I get a fair number of errors, of the sorts:
2 questions
Is this my best route? I'd be happy to use another method (e.g. immich-go) so long as I can maintain my albums and existing dedupe efforts
If so, am I missing the boat on postgres restore? Given a backup from a functioning immich, I wouldn't expect these kinds of errors...29 Replies
:wave: Hey @benomatic42,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.Two notes:
- Immich in LXC is not supported by us and we've seen all sorts of random issues at times
- When restoring your DB you need to make sure that you only start up the database, not the entire stack. It looks to me like Immich started up as well here.
For reference, we have docs on this - https://immich.app/docs/administration/backup-and-restore
Fair point I didn't consider, I will kill the other services and retry.
No change with other services off - is there an expected world in which key constraint violations would show up when restoring into an empty DB? That's what's baffling me so far...
Have you deleted the location postgres writes to, to make sure you're actually starting on blank slate?
Because no there shouldn't be errors about constraints/relations already existing in the DB
I restored from its virgin state, but I can do that as next step.
I hadn't done that yet since the key constraint violations appear on assets -- which AFAIK shouldn't be duplicate in any form of a virgin DB; (and SELECT COUNT(*) FROM assets returns no data).
(the virgin restore begins by dropping all DBs except for postgres and template1)
Oh sorry I missed those constraint violations, I only saw the inserts of existing stuff
We have seen some reports of this happening in other scenarios, too. Those are usually not bad and can be simply resolved by removing the offending rows.
I think I've found the problem - there was an error I missed in the virgin restore where some data stuck around, I'm trying it by hand to confirm.
Confirmed, import successful. @Daniel thanks for thinking with me!
Awesome!
@benomatic42 would you mind sharing all steps for future users? 🙂
I'll see what I can bring into guidance - the specifics were quite messy on my side as I was editing the SQL and retrying in loops while I discovered my own mistakes 🙂
@benomatic42 try your best, I'm in the same boat rn 😅
Ok, here we go. It's rough and ready, and probably wrong.
_
Abstract Description of Migrating docker -> proxmox (community scripts) immich.
Note: I fumbled through this, and my actual path was longer and dumber. I just scratched this out to share what should work better.
Requirements:
- docker & proxmox are running identical Immich versions
- handy enough with SQL to tweak as needed
Backups:
- stop all immich services except postgres on both (via docker stops + systemctl stops)
- make a postgres backup on docker: https://immich.app/docs/administration/backup-and-restore/#manual-backup-and-restore, name it e.g. “to-restore—clean.sql.gz”
- make a postgres backup on immich, same technique; name it e.g. “proxmox—clean.sql.gz”
- make a safe copy of both someplace
Confirm that proxmox virgin restore works, my command was this:
- sudo -u postgres psql --username=postgres --dbname=postgres < immich-virgin.sql
- confirm no errors during restore, maybe confirm 0 rows in some tables
[p2]
Remove postgres user bits from to-restore.
Because a user cannot remove herself, simplest thing is to run everything as postgres user, and only restore the bits now owned by the ‘immich’ user.
To that end:
- unzip the docker backup, remove the ‘—clean’ from the name, and edit as you please
- remove sections (these occur in order in my backup):
- DROP DATABASE immich
- DROP ROLE postgres + subsequent CREATE+ALTER of postgres role
- entire
template1 section
- in my case, this took me all the way to “— PostgresQL database dump complete”, and follows with “Database ‘immich’ dump”
Update ownership: in my case all the “immich” database elements were owned by postgres user, now need to become immich user.
- find each matching “OWNER TO postgres”, confirm it’s part of our immich structure and update to “OWNER TO immich”
- in my case this was 60 ownership updates (grep makes it easy to count)
Now try to restore, and search hard for errors!
- my actual experience was worse than this, so I had to make some manual tweaks to catch and correct my own errors
- if restore fails, track it down, figure out what to tweak and restore the virgin database before each attempt
- as mentioned above, duplicates/key conflicts mean virgin restore isn’t working!
there are undoubtedly better ways (eg, change ownership to all things in immich db can probably be done with some sql metadata queries...) but I got it working 🙂
I put it above, happy to help if I can although I'm off to bed shortly.@benomatic42 Thank you so much. got it up and running thanks to you 🙌
Awesome! Any surprises or new challenges?
I was using postgres 15 so I had to downgrade for the backups to work. Buts that's all
@benomatic42 Thank you for the description. I'm currently in the process of migrating and will follow your instructions.
@Daniel Is it not recommended to run immich in LXC?
It is not.
Proxmox recommends against running docker inside LXC, and we only support docker, so the Immich proxmox LXC script isn't supported in that sense either
@Daniel @Tower this variant uses a native build instead of a docker container within the LXC: see https://github.com/community-scripts/ProxmoxVE/blob/main/install/immich-install.sh#L161 and the following 100 or so lines for the actual build
We don't encourage running native Immich because we don't support it, you're on your own if you do it. 😉
Which I was saying too
and we only support docker, so the Immich proxmox LXC script isn't supported in that sense either
Docker in LXC is not recommended, especially for beginners like me.
I've read that several times.
But how can I tell from the helper script that this is a Docker container within an LXC?
Sorry for the not-quite-appropriate question here...
You should probably read scripts and understand what they do before running random stuff on your hypervisor tbh
so the answer is, read the code of the script
But how can I tell from the helper script that this is a Docker container within an LXC?If it's not that wouldn't be supported by us. We only support Docker installs That's what I was trying to say
It's not a docker in LXC
It's native immich in an LXC 👀
It’s native isn’t it :hyperkek:
(Which is actually worse for newbies)
Okay, that was a misunderstanding.
I was wondering why Docker was running in LXC, but now I get it.
@Zeus I generally agree with you.
But if I always studied exactly what was happening before trying something new, I probably wouldn't be able to get much done.
I have a lot of trust in my backup concept and the community here 😉
And with every new experiment, I learn something new 🙂
thank you