restart duplicate detection?
Hi, i have >70k files, many of which had fd up date data due to google photos download, which i cleaned up beforehand. Now, after enabling upload from my phone, all those same files have been uploaded without duplicate detection seemingly, and now i have 30k+ files in october 2024.
It seemed to me like the duplicate detection of already uploaded files has stalled or did not work anymore, which is why i hit "clear", which turned out to be a mistake as i cannot find a way to restart the duplicate detection job. Ideally, i would want to re run it for all existing files.
20 Replies
:wave: Hey @dgkz0idberg,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.information:
I am running the unraid apps store version of immich from imagegenius' repo.
i am also running a separate docker immich-machine-learning.
everything has worked perfectly, duplicate detection was flawless. The issue is that i want to rerun it.
If you hit pause, you can then clear and restart it
hello, that is unfortunately seemingly not true in my case, see images attached. Before i orignally hit clear, there were thousands still pending.
Try bringing down the stack and destroying redis and its data, then starting back up
redis is only a db for impermanet data right? nothing breaks when destroying it right?
Ofc, otherwise I wouldn't suggest that
okay ill try that later, right now i am finishing up uploading the last few thousand files from my phone.
thank you for now-
If you're actively uploading them duplicate detection is running as part of that
alright, everything is uploaded. I have watched closesly the jobs page, and for a long time, no value haves changes here, as in no progress is being made for face detection, metadata scraping etc.
@bo0tzz by destroying redis, do you mean removing the redis container, or going in via gui/CLI and deleting data?
If none of the jobs are progressing at all then there's something wrong with your installation
is it possible that theres some overwhelming involved due to 70k+ new files having just come in via the android app?
would evacuating the redis db help feasibly?
after waiting some days and restarting the stack, all jobs now run again. Whats weird though is that currently, theres more than 400k waiting in the duplicate detection job, with that number growing every few seconds by about a thousand.
i don't even have 100k files. whats wrong here?
Could you post your compose/env
You might have nested folders
I'm not with the whole unraid setup, which I assume this is, but I don't think you can put import inside of /photos
I specifically changed it away from /appdata because that lives primarily on the cache which i didn't want to get too big
It all worked fine before i turned on sync from my phone.
Before that i had all files up until 2024 manually put into the /Photos/immich/import folder and it worked perfectly. Now that the phone synced it basically imported all files 2011-2024 again, as well as the missing 2025 files.
Do you have an iPhone
Android
I think #unraid will know better but I'm pretty sure you just made a recursive folder setup like this
Alright thank you. This is my third attempt at running immich via unraid docker.
I think running a clean installation on a vm will be the way to go.