Immich•6mo ago

Duplicates uploaded to album via browser are stored in the filesystem anyway

Running on: UNRAID / DOCKER
Version: 1.135.3
DB: Postgres

Hi folks,

So I deployed a second copy of immich and migrated several libraries into it (my Google Photos had better metadata and I wantd to start with those files first). I used several methods to populate the second instance including:

Immich CLI
Immich Go (which uploaded the content but failed to create most albums beceause they had : is in the name)
Uploading Via Browser DIRECTLY into album

I had a particularly bad time with recreating the hundreds of albums from my 400,000 photos. I originally started uploading the critical albums via the browser (downloaded album by album) and uploaded

It appears by the storage being used (5TB previously to 7TB now) that immich may be storing the photos uploaded via browser that were ALREADY stored on the server, it found the duplicate, and added the on-server copy to the ablum. The amount of extra storage I'm seeing is approximately what I would expect from those rejected duplicates.

It is possible at all that immich is actually failing to remove those duplicates? I know it has to fully upload the file to get the checksum (in the browser). Has anyone experienced this type of behavior? Any idea how to solve it?

Thanks in advance.

Immich•7/13/25, 5:29 AM

:wave: Hey @yensid152,

Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.

References

Container Logs:
```
docker compose logs
```
```
docker compose logs
```
docs
Container Status:
```
docker ps -a
```
```
docker ps -a
```
docs
Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA

Immich•7/13/25, 5:29 AM

Checklist

I have...

:ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
:ballot_box_with_check: read applicable release notes.
:ballot_box_with_check: reviewed the FAQs for known issues.
:ballot_box_with_check: reviewed Github for known issues.
:ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
:ballot_box_with_check: uploaded the relevant information (see below).
:ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

Information

In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:

Your docker-compose.yml and .env files.
Logs from all the containers and their status (see above).
All the troubleshooting steps you've tried so far.
Any recent changes you've made to Immich or your system.
Details about your system (both software/OS and hardware).
Details about your storage (filesystems, type of disks, output of commands like
```
fdisk -l
```
```
fdisk -l
```
and
```
df -h
```
```
df -h
```
).
The version of the Immich server, mobile app, and other relevant pieces.
Any other information that you think might be relevant.

Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)

If this ticket can be closed you can use the

/close

/close

command, and re-open it later if needed.

IImmich ## Checklist I have... 1. :ballot_box_with_check: verified I'm on the latest rel...

ImmichAPP•7/13/25, 5:31 AM

Successfully submitted, a tag has been added to inform contributors. :white_check_mark:

aviv•7/13/25, 10:52 AM

Unless you use external libraries (and according to what you wrote you don't) then immich will always reject an identical file.
You wrote that you are importing your Google Photos library, it is possible that you used a quality setting of "Compress items" and somewhere else on your computer you had the original quality (i.e. files that look the same but differ in their hash values)
Is this possible?

Aaviv Unless you use external libraries (and according to what you wrote you don't) th...

yensid152OP•7/13/25, 2:58 PM

No external libraries. Did not compress any items.

So it's very possible. When I setup immich originally, I imported from Apple Photos, which didn't have a complete copy of all my photos, actually far from it. Google was the best source. I do have a lot of RAW files that came over from Google Photos that are about 25 mb each, so:

1.) It's expected that there will be new files;
2.) I uploaded Google Photos to SERVER 2 with all .json files first and processed everything (including smart search and faces) before adding additional files;
3.) When using immich-go for the Apple Photos library, I only saw about ~1,500 "quality upgraded" photos;
4.) Apple Photos does not have a good date/time base. Many, many photos will have been duplicates with different quality/dates/times -- immich seemed to do a good job rejecting those and the hash SHOULD have regardless of those small issues;
5.) I did check the logs when using the browser and confirm that immich does CREATE an image and when it rejects the duplicate, there are logs for each rejection showing deletion of the asset;

ONE IMPORTANT THING - I did download a lot of multi-part zips from large albums (some as many as 600GB per album). I saw that immich creates the zip and steams it right away, and is suppose to place it in /tmp. Is there any chance those files are stuck somewhere (docker volume, another temp directory on the file system, etc.)?

Yyensid152 No external libraries. Did not compress any items. So it's very possible. When...

aviv•7/13/25, 11:23 PM

1.) It's expected that there will be new files;
2.) I uploaded Google Photos to SERVER 2 with all .json files first and processed everything (including smart search and faces) before adding additional files;

Can you specify the way you did it? If you uploaded these photos via the zip file you got from takeout without preprocessing (such as using immich go for uploading or using GooglePhotosTakeoutHelper) then the hash value of the file will be different because using GooglePhotosTakeoutHelper "burns" the .json data into the exif of the file and thus changes its hash compared to uploading directly via the zip file you got from takeout (I assume immich go also does something similar in the context of "burning") This could explain the reason for the lack of rejection of identical files.

3.) When using immich-go for the Apple Photos library, I only saw about ~1,500 "quality upgraded" photos;
4.) Apple Photos does not have a good date/time base. Many, many photos will have been duplicates with different quality/dates/times -- immich seemed to do a good job rejecting those and the hash SHOULD have regardless of those small issues;
5.) I did check the logs when using the browser and confirm that immich does CREATE an image and when it rejects the duplicate, there are logs for each rejection showing deletion of the asset;

Can you attach the logs?

ONE IMPORTANT THING - I did download a lot of multi-part zips from large albums (some as many as 600GB per album). I saw that immich creates the zip and steams it right away, and is supposed to place it in /tmp. Is there any chance those files are stuck somewhere (docker volume, another temp directory on the file system, etc.)?

immich does not save the zip files anywhere because they are created in memory
https://github.com/immich-app/immich/blob/b0098d6d238fbd22f90cb01d3aa7bf75287fc8f9/server/src/services/download.service.ts#L86

Aaviv > 1.) It's expected that there will be new files; > 2.) I uploaded Google Photos...

Immich•7/13/25, 11:23 PM

    const zip = this.storageRepository.createZipStream();

    const zip = this.storageRepository.createZipStream();

Aaviv > 1.) It's expected that there will be new files; > 2.) I uploaded Google Photos...

yensid152OP•7/14/25, 2:56 PM

I did import all the Google Photos takeout zips with immich-go.

Now that all the duplicate search has been done, I think you're right. immich-go seems to have created duplicates. I see 68,000 files that are tagged as duplicates in the duplicate utility, so. that is most likely my explanation.

I noticed that there are a huge amount of video duplicates.

yensid152OP•7/14/25, 2:57 PM

yensid152OP•7/14/25, 2:57 PM

so it seems I'll need to go through all the files and dedupe everything manually (go me)

yensid152OP•7/14/25, 2:58 PM

and thsu beging the next several weeks of my life

yensid152OP•7/14/25, 2:58 PM

QUESTION: In terms of duplicate videos, my understanding is that immich skips 10 seconds in and does a grab for the thumb and that the duplicate search would be based on the thumbnail, am I correct?

yensid152OP•7/14/25, 2:59 PM

(really appreciate the very clear assistance btw)

Yyensid152 QUESTION: In terms of duplicate videos, my understanding is that immich skips 10...

aviv•7/14/25, 9:42 PM

Yes it has been happening since #9784 thanks to our ML wizard

https://github.com/immich-app/immich/blob/5f10a4cae7c24c468521362e1fd9189f9b4648ca/server/src/utils/media.ts#L407

Aaviv Yes it has been happening since #9784 thanks to our ML wizard 🪄 https://github...

Immich•7/14/25, 9:42 PM

      'thumbnail=12',

      'thumbnail=12',

[Pull Request] feat(server): better video thumbnails (immich-app/immich#9784)

Yyensid152 and thsu beging the next several weeks of my life 🙂

aviv•7/14/25, 9:44 PM

Yes, it's terrible when there are so many sources and then things like this happen, at least now you control your files and don't have to depend on the kindness of big companies to create an archive file of your information for you without using any additional tools to make it truly meaningful.

yensid152OP•7/14/25, 10:37 PM

I was about to get through 8,000 files quickly, so it won't take THAT long to sort through things. Putting my Unraid to good use.

Actually I'm very very happy which immich. In many ways, it's better than Google Photos, and I like the control and backups. Really excited to see the reworked mobile apps with better syncing as well.

Thanks again for all your help.

Duplicates uploaded to album via browser are stored in the filesystem anyway

References

Checklist

Information

Similar Threads

Similar Threads

References

Checklist

Information

Similar Threads