UPLOAD_LOCATION folder on host doesn't contain Immich files
As in the title, the folder specified in UPLOAD_LOCATION env var seems to be ignored, as though it's not mounted.
/usr/src/app/upload within the docker container does contain the files, but that is not reflected on the host system. I manually copied the contents from within docker to that location, but it is clear that it is still not 'connected'.
I'd love some pointers as to where I might have gone wrong.
Relevant config/terminal output:
docker-compose.yml
.env
Exploring the contents of the docker container
External drive info
sudo docker system df running but is slow so pending results.
Immich seems to pick up the right drive information in the UI: 3.3 TiB of 3.5 TiB used
I previously had the env var mounted to a different path (which is a subfolder within my user directory and Immich would instead provide the disk usage of that drive (500gb, 80gb of which is Immich files). Using the path that the drive is actually mounted at did seem to fix the disk usage stats in the UI but hasn't otherwise fixed it.
Anything obvious that I might have missed?40 Replies
:wave: Hey @Smilebags,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.Can you share the output of
It shows the standard folders because I just ran a
docker cp a few hours ago.
Did you change the env file before or after building the container?
How are you planning on troubleshooting this then?
What I have confirmed is that adding a new file and navigating to the relevant path based on the files UUID shows that the necessary folder is missing.
Until today, the folder was completely empty
I
docker compose down and docker compose up (without -d while debugging) after changing .env
So I can see the server logs. I have tested whether it's permission related (by running the containers as a different user), it isn't, they're running as root, when running as another user, I get permission errors in the logs.
I'm happy to change UPLOAD_LOCATION to a new path to help with debugging.Can you share your whole compose and env file? Feel free to scrub prostgres credentials
When you run docker compose down, what's the output of docker ps before and after?
Before
After, empty (only contains unrelated containers)
What about docker ps -a
Still after running
down
docker image prune
When the container is stopped might be a good idea
Is this Ubuntu perhaps
Yes
Is it snap docker?
Can you tell me an easy way to check?
I'm running it headless, docker has been setup for >2y
sudo ps aux | grep snap perhaps
Might be -aux I'm a bit rusty
Doesn't look like anything docker related in there
How confident should I be that a prune won't nuke whatever copy of the data the container does have access to @Tempest ?
OK probably not snap docker then, because the volumes for snap get put somewhere weird which would explain this behavior
I thought you said that you had a backup of the data?
I have a backup of that folder yes, less confident about the DB, or any unknown unknowns.
Because right now we ARE trying to nuke the data, so it uses the correct file paths
What's the output of
docker inspect immich_server?Nothing when it's not up. One sec, spinning it back up
Sorry not nothing.
The mounts there look fine hey
Yes, so it is mounted to the container right, and
Immich seems to pick up the right drive information in the UI: 3.3 TiB of 3.5 TiB usedsuggests the same So then let's scrutinize the original issue
it is clear that it is still not 'connected'.How are you determining this?
Previously, I was determining it because the folder was entirely empty. UPLOAD_LOCATION was previously
~/share/skyhawk/immich which resolves to the same folder on disk, but I imagine that's what was confusing Immich previously regarding the drive capacity.
The way I'm doing it now (though I can do something more heavy handed if needed to confirm) is to upload a new image, and then look for the folder path in /skyhawk/immich/upload/6872c15a-dbad-4bbc-952d-5b9689e6f9d4
Images that existed before I docker cp ed them over, are obviously there. New images I upload don't show up.
I can rename a folder and restart if that would helpDo you have storage template enabled?
No
You say previously it was pointing to the same folder on disk through a different path - how was that set up? If it was working, why was there any need to copy files over?
They weren't showing up in that location (either /skyhawk/immich or the user folder subdir). It was "working" to the extent that Immich had the files somewhere. But I don't want my files 'somewhere' that I don't know, understand or have control over.
I suspect they're stored in a docker container somewhere, or somehow mapped to a different location on my main drive.
Right, so it was never really working properly.
Tbh I have no idea why your docker setup seemingly doesn't mount volumes properly
That's relieving to hear in a way 😆 It has completely stumped me.
I might further test my assumption that it's not mounted correctly by renaming a file from docker and seeing whether it updates on disk. That feels like the weakest part of my understanding here.
If you create a new immich instance in a new directory, does that work as intented?
I might try that next.
I appreciate the help everyone! I'll loop back if I find the root cause
I definitely did myself a disservice by copying files over and simultaneously tried to debug the issue
Though good news is that it looks like it is indeed linked now. I created a dummy file from the server container and it showed up on disk so.. It looks like it is indeed working???
Now to find 80gb of zombie files
My assumption equating the ID for an image in the FE (page URL, image URL) to the ID used for file storage may be wrong, suggesting why I couldn't find the new files. They may exist, somewhere in there, with an ID I don't know how to find
Odds are those ended up inside of the container's temporary fs and then got deleted when the container was recreated
I've restarted immich multiple times prior to today
Though that sort of thing is the concern that motivated me to try to sort this out
I don't want some random docker container owning all my files and deleting them with an accidental command
If they are in a docker volume I think you should be able to find them in var lib with
sudo find /var/lib/ -name somefile.jpgAlso try
docker volume ls