Problems after truenas update
Hello everyone,
Lets see if I havent messed it up completely...
Recently my phone app got updated (now is showing a beta symbol when I open immich), (an iphone)
After this update, I wasn't able to see my pictures, I did a quick search and found that there could be incompatibilities between the app and the version of immich I was running in my truenas
Then I tried to update my app in truenas, I also got another error due to unable to execute a change in some of the fields (string value vs freetext value or something similar) so I couldn't,.
Then I decided to uninstall completely immich and try to install it again and reconfigure it, nope, I wasnt able, again another error which pointented out to problems with compatibility with my truenas version
So then I decided to update my truenas (with a couple of hiccups), now I am running 25.04, I was running 24
Re-install immich and perform the migration of datasets (I do believe I have do it properly) as I was running with the old system with multiple datasets (upload, backups, library, video...)
The issue now is whenever I reconfigure immich to use the new dataset I created, as soon as I click start the app, enters in a loop of deploying/running
When I click to launch the web UI, I got an error of not possible to connect to it
What can I do? how should I troubleshoot it?
I am not that expert in this area
291 Replies
:wave: Hey @SelfResetInput,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:On the TrueNAS apps page, when starting immich, there is a workflow widget with « server »
Could you try to get the logs of it?
@Xiticks there you go, seems to have problems with privileges
Can you share the permissions you set on the datasets?
the data and the postgress one ?
not sure if this is what you need, if not let me know exactly what you are looking for
No, if you click on the dataset, there should be a widget "permissions"
right to Dataset space
there you go
That is the one of the immich dataset, I'm interested in the 2 children so
data and postgres-data
I've stuff to do this afternoon, but I'll send what it should look like afterwards, and how you can correct it
they look the same, thats why I put the parent
but now that you mention this, I remember there is also a owner of the app that you can change, has something to do here?
oh yeah was not obvious to me haha
Yes, exactly
well not this way, I'll show you what it should look like
But first, on Immich dataset, based on that
change acl type to nfsv4 and the acl mode to passtrough
You can also do it for data and pg data datasets
then on data dataset, set the permissions so they look like this (and TICK THE 3 boxes, 2 up apply owner/group but also the apply permissions recusively. Do not tick apply to child dataset)
As for pgData you set them using Immich
You edit your Immich app, an tick the automatic permission
Looks like I cannot even do this, I am getting an error when trying to change to nsfv4
I will create another dataset and try to replicate the setup considering your steps and lets see if I have luck
you started with the Immich dataset? If so that is indeed strange
Hello again
I configure the datasets (new) in the way you told me and seems it is still not working properly
They’re not correct
They should look like that
like you need to add the User -apps and other stuff
got it, I thought that was custom from you, let me try again
🙂
now time to synch/move from the older location
any recomendation on how to do it?
I guess, first synch in case it breaks again
btw, is it possible to restore the setup that I had? I have it with my wife as another profile and photo sharing
Wait what you should not have that without your photos
the dataset is empty
now I need to move from the old dataset to the new one I created
Should have done that the other way around
This won’t work for the Postgres data
whops
so I need to do it again
but before launching the app, migrate all the data
Yes
Well
or I guess I can put all the previous photos as a external library?
No
We will sync them afterwards
For now just recreate datasets for pgData and data
https://discord.com/channels/979116623879368755/1178410588821524561/1421124186977337414
Like in there
I’ll be more available later
Wait
stop
You alrady have that
How did you setup your new Immich install?
you mean paths?
Yeah
Don’t delete anything of your setup
Why is that immich1 ?
You had immich there
It is the new dataset that I had to create, the previous one I couldnt change the type of ACL type
or yes
wtf
You can change the type
I was getting an error
let me try again
anyhow, maybe I need you to confirm me if I have properly setup the original dataset
when I compare sizes
it does not match
thats the original, from this one I "moved" files to the Immich
and then
so is there a way that I can compare if I have files in the old dataset that are not in the new one?
so I can move those
Have you enabled spashots in the past on your old datasets?
Otherwise doing the rsync again
maybe, how can I check it?
but, rsynch with the main dataset?
No
or I need to do all the old datasets
I’m doing other stuff now
to the data
Is that possible for you in 2 hours maybe ?
Or tomorrow
tomorrow better
you are the hero!
tomorrow same time as today?
Like same hour as now ?
Yes can do
earlier I am also available
Tell me your time and time zone
(UTC+02:00)
Oh I'm on the same so it's fine
20h00 is fine for you or are you available earalier?
20 is perfect
thank you sir
talk to you tomorrow
@SelfResetInput If you're already there, I'm available
here I am
so, do you want to check anything before doing a rsynch?
oopos
sorry did not have notifications
no worries
So to recap, you already tried with rysnc and you get dataset with less storage
can you share the whole UI page, when you click on this upload dataset
with all the dataset info
the very first time I did a "move"
and den I could see that videos dataset was still there with storage, so I did a rsynch for that one
yes, give me a sec
wiat but that seems weirs also, like your video dataset is empty
still show me the details, but I think I undersand your issue
So after showing me that, inside the trueNAS shell, cd into your old upload dataset and do
ll once you're in it
and share the outputI am back
had to "hard reset" the machine
so, in order to clarify
Immich -> the dataset I created originally for the new version and where is supposed to be all the data already
immich1 -> dataset I created after facing problems with immich not launching
inmmich -> My old dataset
Ok
Can you share
1) Your current Immch storage config (like when you edit the app)
2) The content of the upload folder in your old install
the immich config I am suppose to change it
Oh you already changed stuff, that's right
Well I need, that
so in the TrueNAS terminal
are you sure? it shouldnt be Immich? instead of Immich1?
No I mean you already disabled the old config
yep
it looks like something was already not correct before
that's why I want to see the content of your old immich datasets
then do
ll
(Inside TrueNAS shell)
and if, from there, you do
cd ../video
then ll again
Mmh, ok, that's not normal
but we will try without that
so can you delete data and postgres-data and recreate them?
use the "apps" preset for the data one
and leave the "generic" preset for the postgres one
you mean from Immich1?
Oh yes
I mean the dataset you'll use for the new install
I want you to start fresh again
give me a sec
deleted
I guess you refer in the immich config path for data storage?
Yes?
Like create 2 datasets, like you did before
noted
one that will be used by immich for the data
and the other for the database
now I got you, I was confused, I can see what you refer with generic
done
can you show their name?
like the truenas gui with them
this seems correct?
yes don't change it
ok so now
from terminal
run each of this command line by line
got it
executing last line
yeah
it will take a while
should I also do a rsync with the dataset where originally things were moved?
aka immich?
Mmh no ?
What do you mean moved ?
No you don’t need to
as mention, the very first time that I tried to update, when executing the commands of transfering the data, I used one that moved the data instead of synch
As inmich is the original
Oh
That explains it
Holy shot
🙂
yeah, lesson learned
Can you cd into the immich folder then
And do ll inside it also ?
Well after the initial rsync is done has
I guess I need to wait for the rsync or I can open another terminal?
Yes
Just wait
got it
how can I pay you for a coffe
or a beer
whatever you like
already cd into cd /mnt/CrisHaiz/Inmmich/upload
No I need you to go in Immich
Immich/data more precisely
yeah, sorry
I am already in the data dataset
yeah
then do ll and share it
ll
I forgot about it
haha
so do tehse
for all folders that are in there
sudo rsync -av /mnt/CrisHaiz/immich/data/upload/ /mnt/CrisHaiz/immich1/data/upload/
sudo rsync -av /mnt/CrisHaiz/immich/data/thumbs/ /mnt/CrisHaiz/immich1/data/thumbs/
sudo rsync -av /mnt/CrisHaiz/immich/data/profile/ /mnt/CrisHaiz/immich1/data/profile/
sudo rsync -av /mnt/CrisHaiz/immich/data/library/ /mnt/CrisHaiz/immich1/data/library/
sudo rsync -av /mnt/CrisHaiz/immich/data/encoded-video/ /mnt/CrisHaiz/immich1/data/encoded-video/
sudo rsync -av /mnt/CrisHaiz/immich/data/backups/ /mnt/CrisHaiz/immich1/data/backups/
🙂
how do you put code in the chat?
three `
before and after
thanks!
I guess will take some time
done
ok so now show me immich1 datasets
does it report the same amount of storage as befroe?
seems that now the big one is immich1
ready to launch?
Yes?
Looks like it
'''Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 515, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 562, in run_body
rv = await self.middleware.run_in_thread(self.method, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 627, in run_in_thread
return await self.run_in_executor(io_thread_pool_executor, method, *args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 624, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 101, in wrapped
result = func(args)
^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/app_scale.py", line 62, in start
compose_action(app_name, app_config['version'], 'up', force_recreate=True, remove_orphans=True)
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/compose_utils.py", line 61, in compose_action
raise CallError(err_msg)
middlewared.service_exception.CallError: [EFAULT] Failed 'up' action for 'immich' app. Please check /var/log/app_lifecycle.log for more details
'''
```Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 515, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 562, in run_body
rv = await self.middleware.run_in_thread(self.method, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 627, in run_in_thread
return await self.run_in_executor(io_thread_pool_executor, method, *args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 624, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 101, in wrapped
result = func(args)
^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/app_scale.py", line 62, in start
compose_action(app_name, app_config['version'], 'up', force_recreate=True, remove_orphans=True)
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/compose_utils.py", line 61, in compose_action
raise CallError(err_msg)
middlewared.service_exception.CallError: [EFAULT] Failed 'up' action for 'immich' app. Please check /var/log/app_lifecycle.log for more details
```
sudo tail /var/log/app_lifecycle.log
In TrueNAS shell
give me a sec
I saw a difference in the name of the dataset
data vs Data
nope
show me your Immich config
storage part?
could be an incompatibility between old version?
Did you click with the tree to select the dataset ?
nope
let me try that
Otherwise delete the "old" app and install a new one
same error
let me try that
images also?
No
Uncheck them
forget it
I saw it
number of cpus
but then again
let me do again sudo tail /var/log/app_lifecycle.log
looks the same error?
Did you check automatic permissions when you installled the app ?
nope
let me try again
now I did and the same...
should I mark enable ACL?
Mmmh no
if you cd into /mnt/CrisHaiz/immich1/data/backups/
and do ll, can you share the output?
wait a minute
can you do that for all in other folder ?
even in postgres-data
ok and inside this, can you cd into the backups again and do ll again,
no
you're in postgres-data there
not in data/backups/backups
cd into backups again
And do ll there
cd /mnt/CrisHaiz/immich1/data/backups/ or cd /mnt/CrisHaiz/immich1/data/backups/bakcups
The second one
Mmmh
looks like your latest backup is from the 16th of september
Is that possible?
Has Immich runned since?
I dont think so
Or at least do you have the originals for the photos between the 16th and today?
I just moved to a new house
so I didnt have too much time to focus on this
oh so your nas might have been down for a few days?
I do have the originals yes
yep
Ok, then we will rollback to there
will be way easier
when you are supposed to create the new directories.. are these just directories that you create with mkdir within the dataset or are these datasets within the dataset?
but first run
you're not suppose to create any new directories, here its because I misunderstoof what he did
im just trying to read through and help myself here.
ping me in #truenas channel for questions or create another ticket please
ok
is the output what you expected?
no 😄
can you do
ls /mnt/CrisHaiz/immich1/data/encoded-video/
what the actual shit is going on
hahahaha
rsync -av /mnt/CrisHaiz/immich1/data/encoded-video/video/* /mnt/CrisHaiz/immich1/data/encoded-video/
try that insteadcould be that when we did the second rsynch fucked it?
or overwrite it?
???
are you trying to synch the old dataset?
well, here you have video
but it should exists, so I need to move the stuff inside video into the encoded-video
and no
can you
cd /mnt/CrisHaiz/immich1/data/encoded-video/video/doneç
do
ll inside it
ok so that is fine
now do
cd ../ && rm -r video
nice
now do cd ../backups
then
mv backups/* ./
should I get any message?
no
ll now
ok so again rm -r ./backups
then cd ../library
and ll again
(it will change at some point so just wait my indications)
ok so same here again
mv library/* ./
then ll
mv?
?
can you
cd library
then ll
ooh makes sense then
cd ../ && rm -r library
share ll again right after
just to double check
Ok, so now
cd ../profile && mv profile/* ./
then again ll
now
rm -r profile
and for our final candidate
cd ../upload/upload && ll
and show the content
ok so there, run
rsync -av ./* ../
ok so now
cd ../ && ll
nice
now
rm -r upload
this way we fixed the "data" datasetI trust you 😄
so now, click on postgres-data in and share the whole ui
click on edit next to permissions
make it look like this
(with the boxes checked and unchecked everywhere, it should be exactly the same)
you forgot apply user
apply user
marked
click on save then
saved
Now, try to install Immich again
deploying...
if it doesn't not fail instantly, it might be a good sign yes
still deploying
click on the third button on the first line
so looks good!
and share the Immich logs
that's the bottom of the otput ?
lets go!
can you just double check the logs to be sure
server logs?
from my phone, it has connected again
and I can see pictures
like these
the screenshot seems cropped
and also, what happen if you upload a random screenshot from the WebUI ?
I just want to be sure that it works
should I launch again the app?
No, no
That is fine
just try this so I know that everything works
let me try it
maybe, I should give it some time
to recognize everything?
https://ko-fi.com/xiticks
or
https://paypal.me/xiticks
because you asked
mmmh
no
here
if you click on the three dot menu upper riht
then click downlaod
does it download the image?
yes
and is that the random screenshot you uploaded as I asked ?
or is is another image?
another image
today's screenshot doesn't appear at all?
the one I uploaded is there
Oh ok, so that is fine
yes, I selected a picture with an older date
I had to search for it
it's just an issue with the thumbnails
https://my.immich.app/admin/jobs-status
bug or app just need time?
so go in there, and run the first job on "all"
migration bug, but easily fixed
The screenshot upload was to check that we did not have an issue
and the download was to check if immich still knew where your images were
Can you also run the third from the bottom on all?
48k thumbnails to be created
the "transcode video" one
"re-created"
be yeah, it will take a while
just maybe try to download some random images/videos from different time period to see that you have them all
and also try to take a picture on your phone to see if it's also uploaded
fom the webui
I have to download to see the picture
yeah, when this job is done, you'll see them directly in the webUI
Immich show thumbnails not the originals (ensure better compatiblity)
yes, I had to enable backup from the phone
and I can see that appear in the webui
Nice, so everything looks like it is fixed and work!
Just have to wait for the jobs to run to see images in the webui
will check again tomorrow
thank you so much!
hope that next update goes better
The migration is a one time thing in theory
so next time you should be able to just hit the update button (as long as check for update often)
well, if it is decide to migrate to a new setup again 🙂
It should not
Pleasure to help!
I will spend some time also to create proper backups
so in case that immich fails...
Well for starter, you could enable the storage template
do you have any recomendation on how to extract/backup whatever is in immich=
On the other hand, with TrueNAS, the simple solution would be to replicate/rclone the datasets
hmmm
that looks amazing
this way you would only need to save the backups, profile, library and upload folders (as well as the postgres-data)
https://my.immich.app/admin/system-settings?isOpen=backup
Can you also tell me if that is enabled on your side?
ok so it's fine
your database is "backed up" every night into the backups folder
by database, you mean the config?
the postgres-data
It contains the config as well as the information on where the photos are stored, the albums and so on
maybe I bother you again in the following days with some questions
so happy to see people willing to help and support that fast
No worries, but if TrueNAS related, do so in #truenas
Well I've been on a helping spree with the recent "forced" TrueNAS migration
So glad I could help with it
You can /close this ticket
will do
have a good night!
Good night also!
This thread has been closed. To re-open, use the button below.