Storage Template Problems
Hi all, I have setup a storage template and run a storage template migration to update the entire library. I'm bow trying to manually back up my library following the instructions in your docs, but I see a few issues which are concerning:
1) some pictures remain in the
uploads/ directory (only 22 out of ~8k total)
Why aren't all photos moved to my library?
2) mysterious 'admin' user has ~30 photos
My account was previously named 'admin', but I renamed to 'John' and set the storage name to 'john' as well. I also set the storage name for the only other user on my server to 'alden'.
When I look in the library/ directory I see:
Why is the admin user library still there? Why does he only have ~30 photos? I've rerun the migration and updated/restarted Immich several times, but these photos remain here.13 Replies
:wave: Hey @wizeguyy,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:1) How long have you been using immich?
2) The old (admin) folder maybe wouldn't be cleaned up after the storage label rename.
Can you investigate some of the asset names that rename and look and see if they still show up in immich fine? You can look at their location in the file system using the detail viewer
1) I've been using Immich for about a year, IIRC
2) weirdly, it did go away at first, but about a day later it reappeared. The photos in the admin folder are a small (seemingly random) selection of photos from the 'john' user
I will check to see if those assets appear in the timeline
One sec
I did some investigating...
The images in the admin folder do not appear in the 'john' folder, although they were uploaded by 'john'.
The images do not appear in 'John's timeline, although they should.
Nothing appears unusual about the offending photos. Same file naming convention as every other photo
Ahh, upon inspecting each asset left in the 'admin' folder, I realize these are assets that were in the trash at the time I renamed the account's storage name. My guess is the storage template migrating doesn't handle trash items properly?
Solution:
1) Deleting the items from my trash made those files go away
2) running the storage template migration job again cleared up the empty 'admin' folder in my library
I've now confirmed that new trash items are added under the new storage name ✅
@schuhbacca before I close this, I want to make sure you've noted the bug about trash items not being handled properly when the user's storage name is changed
Hey sorry. Been busy with work stuff. Let me take a note down to validate this is addressed. Appreciate you researching into it!
It seems that handling of trashed assets during storage template migration has changed in v1.130
https://github.com/immich-app/immich/pull/16700/files#diff-160ca0c92818fe07b15afe84633d23c96c44bf66c97c54bf6d0d25cdaafebadfL124-R121
- Previously it was getByIds() without filtering out trashed assets
- Then it switched to getStorageTemplateAsset() which excludes trashed assets
@jrasm91 correct me if I misinterpreted the PR
Close but incorrect
At the time of that commit we used
getAll to queue assets https://github.com/immich-app/immich/pull/16700/files#diff-160ca0c92818fe07b15afe84633d23c96c44bf66c97c54bf6d0d25cdaafebadfL157. That directly uses the search asset builder and since we don't pass withDeleted: true trashed assets will be filtered out too https://github.com/immich-app/immich/blob/main/server/src/utils/database.ts#L397I guess if an asset gets trashed while the storage template job is running, the behavior has changed, but it's (IMO) more correct now
I see, thanks for the explanation. (I inspected
handleMigrationSingle instead of handleMigration. Therefore wrong conclusion. They have slightly different code/queries - which is expected afaict)handleMigration queues jobs, handleMigrationSingle runs that job
Which is why I was saying that you could in theory run into differing behavior if assets were trashed between the job queueing and the executionGot it. So subtle differing behaviors are possible due to concurrency:
- (a) Trashing asset between the job queueing and the execution
- (b) Restoring asset from trash between the job queueing and the execution
Of course nothing critical here, just an asset may be left in the old folder. Then, supposedly, second run of the Template Migration job can fix it.
In the scope of this ticket, assets were trashed before enabling Storage Template.
So they are ignored by the migration. They are left in the old directory.
Their destiny depends on whether the user restores them or abandons them for good.
- If no one cares about trashed assets, they will be gone in 30 days. No files left. Maybe only a few empty subdirectories will be lingering.
- If the assets are restored, they will live in the old directory. Then next manual run of the Storage Migration job will move them to the new/proper location. Right?
It seems there is no real impact. Unless the user removes the old directory and then decides to restore an asset from trash. But it's already gone.
I'm just thinking whether this is considered a bug (then we should migrate trashed assets as well) or somewhat expected behavior.
We don't queue these jobs individually. There is a bulk job and a single job
The bulk job does all of them at once