Issues during Storage Migration on TrueNAS Scale
After doing the storage migration following the official documentation my server would not start. I would have needed to upgrade to 1.143.x to fix this but as I was a little late to the party, that was no longer provided by the official repository. I foolishly followed the instructions linked in the "Invalid upgrade path" error and tried to downgrade to 1.132.3 using the snapshot. I rolled back to 1.135.0 since I was unable to find the correct version but now I can't roll "forward" to the snapshot of 1.142.1, which I had been using previously. I would need some assistance on going back to that and afterwards upgrading to 1.143 for the migrations so I can upgrade to the newest version. Setting up a new App instance didn't work since the necessary migrations are missing, if there is a path to do that without data loss it would be great too. Thank you so much for your help in advance!
142 Replies
:wave: Hey @miasinterestinglife,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.1 doesn't really apply since I am trying to achieve just that
Immich-Server Pod log:
pgVecto Pod:
machine-learning Pod:
Redis Pod:
(cannot send because it is too long, but indicates a normal startup without issues
Permissions Pod also starts normally but log is too long
Troubleshooting Steps so far:
- Upgrading didn't work due to the unavailable intermediate version (1.143.x)
- New Instance didn't work, again due to missing intermediate version
- accidentally/stupidly rolled back to way too old version (cannot change storage layout now because the dropdown for pgData is greyed out)
- cannot update due to inability to change to new storage layout
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:
@Xiticks
Could you also share your current Immich config? As well as the dataset you have on the system?
It seems that you did not needed to rollback to 1.132.3 or other version as you were already on 1.142.1
Yes, I was very sleep-deprived at the time and just followed that advice without thinking too much
and this is what happens when i try to change the storage confuguration
Ok, so you were using ixVolumes, so your original is somewhat intact,
Which version of Immich is this atm ?
1.135.0 is what I rolled back to but it should be 1.142.2 (or another minor patch, could be 1.142.1 too)
What I would try first, given your explanation
Delete the Immich app
DO NOT CHECK THE BOX "Delete IxVolumes" Otherwise there is no way back
Then you install a "new" immich app and point data to the "data" dataset, and postgres to the pgData one
Already tried that, it still throws the "invalid upgrade path" error
The only installable version is 1.144 (or newer) and I am probably missing some migrations from 1.143
I just got the tip to convert it into a custom app and change the version that way manually, is that safe to do?
Do this anyway
As you will do that
oh, okay
I can guide you through it
all the data should still be in ixVolumes atm, so there is a way to rollback if something else happens
alright just set up another instance pointing to the new datasets
so i convert it to a custom app now?
Check the logs first, just to be sure
yup, it's throwing the error as expected
Ok so convert to custom app
2025-10-02 12:21:22.266282+00:00Error: Invalid upgrade path. For more information, see https://immich.app/errors#typeorm-upgradebut do not change the version yet
okay, so i just click on the custom app button
So edit the version, but try 1.136.0
around line 173, there is
image: ghcr.io/immich-app/immich-machine-learning:vx.xxx.x
change vvx.xxx.xto v1.136.0
Do the same at around line 419
for image: ghcr.io/immich-app/immich-server:vvx.xxx.xweird, all the images are at 1.144.1
don't know what happend there
you installed the latest available
so this makes sense
oh yeah, forgot that part
If you refreshed you apps, it might even have been 2.0.0 as it's now avaible on TrueNAS, but we don't care for now
i appear to have a permissions issue:
2025-10-02 12:29:49.815967+00:00[31m[Nest] 17 - [39m10/02/2025, 2:29:49 PM [31m ERROR[39m [33m[Api:StorageService][39m [31mFailed to create /usr/src/app/upload/encoded-video/.immich: Error: EACCES: permission denied, mkdir '/usr/src/app/upload/encoded-video'[39mOooh
but it has started and done some typeORM migrations
Well, then let it run for now
the server is shown as exited
ok so that's "fine"
did not know this would be necessary
Do that then
it is importing a lot of data and saying the application successfully started
Can you share the logs just to be sure?
But if you can access the webui, it should be good
I can, i am just unable to login
i could try resetting the password to rule any issues with the instance out, not sure if that is safe though
?
How so?
You forgot your password?
Can you log into the server shell
(The Immich server shell)
and type
immich-admin list-users
Just to see if it list at least some usersi just realized what was the issue
I had an older broken immich instance
it's that dataset
For? Everything?
yup, apparently
How did your rsync your stuff ?
i don't know how THAT happened
All from ixVolumes?
or you had datasets on a pool at some point?
like this:
rsync -av /mnt/Data/ix-applications/releases/new-immich/volumes/ix_volumes/uploads/ /mnt/Data/immich/data/uploads/-_-
but that is the new instance
That's probably not
nope, always been ixvolumes
can you cd into what's written in the guide?
first
then
then you do in there and you share me the output
oh
ohhhhhhhh
as well as
and you also share this output
You come from an "old" Truenas version, correct? Like Cobia or Bluefin?
yup
that's what broke the old instance back then
share that alos, please
it's still running
Yes, so when they migrated from k8s to docker, they made a change in the ixVOlumes, but kept the "old" way if some migration did not work properly
As you might have been out of timeline, your apps disappeared, and you reinstalled fresh
So when it broke, I suppose you had to upload everything again?
yes
and you have other apps that you had to reset?
yup
and you also used ixVolumes for those?
yes
because I think it would be save to delete this "ix-applications" dataset
should
du be taking this long?Did you ssh into the server or did it through webUI
and if webUI are you using firefox?
i'm doing it through ssh
oh well, then it might take a bit of time
because it "reads" everything
and you might have a lot of thumbs
yes, there's around 15-20.000 pictures in there
I just want ot verify that your files are indeed there
as if that is the case, you could probably do that
After we set everything
okay i've switch to the web interface now
Don't need to, ssh is better for the command stuff
when the du is done, and the outputs looks like your stuff is indeed there
you can/could remove the data and pgData from here
As they come from the broken instace, right?
It seems weird though, because "data" is bigger than your previous ix-application dataset
so yeah, would wait for that to finish and verify
seems to roughly match, gonna look into the folders to verify
according to the folder changed dates these are as outdated as the other stuff
There is no "backups" folder though
there's both instances there
But it looks like pgData is more recent
okay the change dates appear to be unreliable as subfolders are much newer
yes, that's somewhat what I thought
yeah, i was ini the broken instance, oops
was going to ask you this
i'm doing it for the correct one now
could you jsut check into pgBackup?
Just to see what's in there
because it should have also been a "backups" folder
but it ain't there idk why
(the broken instance)
the new one has an empty (or inaccessible) folder
Wait what do you mean broken instance?
the old one
are you in /mnt/.ix-apps ?
very old one
Or somewhere else?
yes
is there multiple immich ?
/mnt/.ix-apps/app_mounts/
cd into that and do
llyes, there is the instance "immich" and "new-immich"
holy
so
ll into new immich pleash
and show the output
Tahts more like it!
can you cd into backups and also do ll there?
Ok, so that's nice
okay so I must've been on 1.143
So you rsync THESE into data and pgData
following the guide
alright, should i clear those out first?
(no need for pgBackup, it's an ould)
you mean the dataset there?
yes, the newly created datasets with the ancient data
yes, I would even delete them and recreate them
make sure you select the "apps" preset for the data one (as instructed in the guide)
as for pgData, you keep the generic preset
doing that now
yeah, will take a bit, so ping me when it's done
But based on what you said: rollback to version 1.135 or something like that, I think that after the sync, install the app from the catalog would work
because the breaking changes of 1.136/1.137 are handled by Immich given the TrueNAS config
@Xiticks just a quick question: the documentation has the uploads folder called "upload" but in my dataset it is called "uploads", which one is correct?
The latter, you have to rsync from uploads to upload
Previously as it was a dataset, the name did not matter (because it was mounted with the correct "upload" name) but now that it will be a folder inside a dataset, and that you only mount the dataset, the naming is important
For example the "data" dataset, you can set whatever name you want, but the folders it will contain: thumbs, library, upload and so on... have to be "exact" match
oh okay, thanks!
if you already rsyned the data, that's no issue
you can just then mv uploads to upload inside the dataset and this will "rename" it instead of moving everything again
@Xiticks just finished copying everything, the new size of around 150GB makes a lot more sense. Can I now safely delete the old apps (keeping the iXVolumes for the moment of course)?
yes!
And as you said keep the ixVolumes
i am getting this error in /var/log/app_lifecycle.log (i replaced the newline characters with actual newlines for readability)
from what i could find this is because the pgData datasets is not yet owned by the user netdata, changing that now
yep
Wait thoguh
I can share a screenshot if I haven't done it already
oh, thanks for that!
it's especially for "apply user" and "apply perms recursively"
but the screen is the default set by TrueNAS, so replicating it will make sure that it's working
yes, but I would've actually been more wrong about the access modes and the group
You have to strip the acl if you have not the same "view"
I've done that and I'm trying to install now
I've done it like this, but it's still throwing some errors and not installing
could this perhaps be an issue with the parent dataset being owned by root?
this looks right to me at least
can you cd into pgData?
and do ls there??
do i need to move the files to the parent folder (the dataset)?
you rsynced wrong that's the issue
so there do
yeah, the issue was tab autocompletion removing the trailing slash after the last folder
Does it work now?
yes it does, thank you soooooooo much for this!!!
it is now safe to delete the old ixapplicatioins dataset, right?
Should be yeah
maybe double check that you have the same upload/library size
on both
well that is the outdated stuff that hasn't been updated since february. in the now running instance the latest pictures from yesterday afternoon are present
oh yeah
but I was also talking about the content of ixVolumes
yeah, it would probably be a good thing to also delete the ones of the now removed instance
both the old storage config and the new datasets contain folders of the same size.
how would i go about deleting them from the .ix_apps folder? (if that is safe to do as that is taking up a lot of storage)
that's safe
should delete immich folder and it content
should this be any cause for concern?
applies to both "immich" and "new-immich" (the very old and old setups)
Onlu old immich has ix-volumes, right?
yes
then remove the old immich first
are you suggesting the fellow rm -rf?
i have removed all outdated instances of immich
no no
So you only have ONE immich instance?
yes
i removed them from the apps interface without deleting their ixvolumes after i had done all of this migration
to remove any confusion: until about v1.125 the first instance "immich" was used until it broke (don't recall for what reason exactly) and then "new-immich" came in, i just kept the old one in case I needed some of that data (i did not). Now, after all of this migration I have a brand-new instance of immich and I removed the other two from the apps interface and kept their ixvolumes. There is now only one immich
very sorry for the confusion