100s of photos in the "uploads" folder that aren't in Timeline
Why am I seeing 100s of photos from the past few years in the "uploads" folder that have not made it into my timeline?
34 Replies
:wave: Hey @Horkia,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
2. :blue_square: read applicable release notes.
3. :blue_square: reviewed the FAQs for known issues.
4. :blue_square: reviewed Github for known issues.
5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.Likely incomplete upload, especially if you use cloudflare tunnel or a poorly configured reverse proxy
how would i go about cleaning that up? It's a little concerning that there is no record of these errors anywhere.
You can enable storage template, then run storage migration job, after that the left over files in the
upload folder are incomplete upload. In the most recent release, we implemented the mechanism to clean up the incomplete upload filesIt’s not very concerning because the mobile app will keep attempting uploads until success
storage template has been enabled since I started using immich. It works for most of my phone uploads. I even ran the migration job manually. There are still photos and videos in the upload folder not being migrated. They span years and I can't figure out a pattern.
it's concerning because i didn't realize there were hundreds of photos that didn't make it into my timeline. I guess their backed up, but I feel like they could have been lost eventually if I didn't happen to check.
The files left over in the Upload folder are partial files, they can be removed
but they are photos and videos that are not in my timeline or saved anywhere else and I want them
Can you try running storage template migration job?
I have run that several times and nothing changes.
And the file that you think is not in the instance, what happen if you drag and drop them onto the web to upload? Is it shown as duplicated?
this is strange. I ran the template migration job for a 6th time in the past few days and it appears to have processed finally! There are still a couple dozen photos and videos but the photos appear to be partial versions of those which have been correctly filed and incorporated into the timeline.
If I would have deleted the contents of the upload folder this morning (even after running the template migration job 5 times) I would have lost hundreds of photos. Is there a way to monitor unsuccessful template migrations?
Hmm 🤔
It should have been solved
Can you continue monitoring this? now that the partial files upload will be removed automatically, please ping me again in a week to see if you still have left over assets in the upload folder
will do, thanks!
I just noticed that the video files remaining in the upload folder have not been migrated. I attempted to upload one of them through the web interface. It uploaded succesfully, but it used the cryptic filename assigned in the upload folder and used todays date instead of the created date which is two years ago.
Are they MotionPhotos/LivePhotos
no, they are videos -- between 36MB and 2GB in size
Are you using CloudFlare?
i have my DNS server set to 1.1.1.1 in synology if that's what you mean
No, what URL are you using in the immich mobile app?
ahh..i use a synology DDNS with reverse proxy
Ok, most likely a reverse proxy issue then
What can I look for and what is the best way to access immich?
Stable connection and bog standard nginx will do the trick
Thanks, I will look into that. But when I click on the button to run the template migration job, if I'm not mistaken, the software is running a task on files that reside on the same device. There should be no need to perform operations across the network.
most likely files that remain in the upload folder are partial files that were interrupted during upload
The photos were complete unprocessed files (that did not show up in the timeline). After running template migration job 6 times, the photos eventually processed leaving behind a few parts of photos that had properly migrated. The videos remaining in the upload folder however, are still entire video files that do not show up in the timeline (have not migrated) despite running the migration job many times.
Everything that stays in upload is not counted as existing in Immich, these were interrupted somehow
There are some partial photos left over and uploads that have successfully migrated. However, there still remains some videos that have not processed / migrated at all. Sounds like there might be some issues with uploading with a poor connection which makes sense. But I would think there would be a fallback in such an instance. Also, the problem appears to be with the migration process and not so much the upload itself because the files were successfully uploaded to the upload folder. They just weren't moved and processed into the timeline even after running the migration job several times. Several videos in the upload folder still aren't being moved after running the migration job.
Rerunning migration won't do anything, they need to be re-uploaded
i'm getting conflicting information here. This is incredibly frustrating. Immich is such an amazing tool but it seems to be unreliable for the most basic of tasks: backing up assets from my phone. This is something that countless other tools do seamlessly, with little effort to setup. Why is this so convoluted and unreliable? I can only backup photos if I have a really good connection? And if my photo upload is interrupted, it gets uploaded to the upload folder, never makes it into the timeline, and i get no notification that there was a problem? This doesn't seem right.
No it definitely retries on it's own until it works, but you're saying it was never uploaded anywhere so then the only solution is to redo it
Also it's a bit misleading to say it fails at basic tasks if it works for so many people across so many install types, and you have a very specific problem
I'm just saying for me it has been unreliable. I was not making a blanket statement that it "fails." I've personally tried many other ways to backup photos and that part has always been simple and reliable. They always fall short when it comes to other features that Immich is excelling at (or at least making great strides toward).
I understand that this is concerning, can you keep monitoring the situation from a clean upload folder?
I am keeping an eye on this as well
I don't know what caused the issue, but it does not seem to be a problem anymore. THanks!