Files Not Moved During "Storage Template Migration"
#### Description of the Issue
I encountered an issue while using the "Storage Template Migration" feature in Immich. Some files were not moved as expected and remain in the temporary upload directory (
- Uploaded files (images or videos) are first saved in
- Some files uploaded during a specific period were not moved to the
- During the period when this issue occurred, there was a large backlog of scheduled tasks for the "Storage Template Migration" process. - Recent uploads are correctly processed, but the previously unmoved files are still not handled. #### Steps to Reproduce While the exact reproduction steps are unclear, the following conditions might have contributed to the issue: 1. Upload a large number of files to create a significant backlog of migration tasks. 2. Observe if any files fail to be processed and remain in the temporary upload directory (
upload/upload/{user_id}/) instead of being processed and relocated to the proper destination (upload/library/{user}/) with the correct naming as per the template.
#### Observations
1. Normal Process:- Uploaded files (images or videos) are first saved in
upload/upload/{user_id}/.
- The "Storage Template Migration" feature processes these files and moves them to upload/library/{user}/, applying the appropriate naming conventions from the template.
- This process works correctly for new uploads, and the files are migrated as expected.
2. Problematic Behavior:- Some files uploaded during a specific period were not moved to the
upload/library/{user}/ directory and remain in upload/upload/{user_id}/.
- Even when manually triggering the migration via Admin Panel > Jobs > Storage Template Migration, these files were not processed or moved.
3. Additional Context:- During the period when this issue occurred, there was a large backlog of scheduled tasks for the "Storage Template Migration" process. - Recent uploads are correctly processed, but the previously unmoved files are still not handled. #### Steps to Reproduce While the exact reproduction steps are unclear, the following conditions might have contributed to the issue: 1. Upload a large number of files to create a significant backlog of migration tasks. 2. Observe if any files fail to be processed and remain in the temporary upload directory (
upload/upload/{user_id}/).
#### Impact
- Files uploaded during the problematic period remain unorganized in the temporary directory.
- Users may have difficulty locating or managing these files due to the lack of proper naming and storage organization.
#### Expected Behavior
All uploaded files, regardless of the upload time or migration task load, should eventually be processed and moved to the correct directory (upload/library/{user}/) with the appropriate naming.
#### Current Workaround
None identified. Manually triggering the "Storage Template Migration" process in the admin panel does not resolve the issue for the affected files.
#### Request for Assistance
- Can you confirm if there is a way to manually process or reattempt the migration for the affected files?
- Are there any logs or diagnostic steps that can help identify why these files were skipped or not processed?
- Could this issue be related to task backlog or a specific migration task timeout?
Thank you for looking into this issue. Let me know if further details or logs are required to assist with troubleshooting.29 Replies
:wave: Hey @Torin,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJAChecklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like
fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.GitHub
immich-app immich · Discussions
Explore the GitHub Discussions forum for immich-app immich. Discuss code, ask questions & collaborate with the developer community.
FAQ | Immich
User
GitHub
Issues · immich-app/immich
High performance self-hosted photo and video management solution. - Issues · immich-app/immich
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:
Are the files in upload/ intact or corrupted? Can you visit one of them in the web interface using the UUID?
The files are fine, they just haven't been moved and named yet.
Can you post the database output of that asset? Do you know how to find that?
https://immich.app/docs/guides/database-queries/#assets
wait me, Wait a minute, I'll research it.
Sorry for the long wait
I found two records. According to the records, the second column is this file.
I checked the logs and found these issues
https://github.com/immich-app/immich/issues/11131
My problem is similar to this post, but it seems there is no solution.
It looks like your database crashed
Yes, but only when a transfer is executed
And after the execution is completed, he will reconnect
Check the database logs for more clues
Ok wait me
Sorry for the long wait. I'm just dealing with some things.
This is the SQL Log
I think your database is corrupted
Seg fault during a query
So is there any way I can solve it?
I would like to point out that I did not manually modify the SQL data before this.
You might need to restore from backup
Try these first
https://immich.app/docs/FAQ/#how-can-i-verify-the-integrity-of-my-database
Ok wait
It seems that
immich.public.exif_city is causing trouble
Yes, this confirms the database is corrupted. Possibly from damaged storage device or power loss
You should restore from backup
If I restore the database to 1/14, what happens to the photos on 1/15?
I seem to have solved it, I deleted the problematic row