Android App - Duplicate local/server copies after gphotos upload
Hey all - I'm not sure if this is a bug or something I can do.
I've setup Immich and been testing it myself and it has been great. I just went through the process of getting my SO's pictures downloaded from GPhotos, uploaded using the mattwilson tool and it all went smooth. On the server, every image and album is there. Great!
The problem arose when I went to then setup the Android App on my SO's phone. I selected the proper folder and tried to do a manual backup. It actually might have worked, but I'm not sure if the amount of "unique" items is supposed to keep dropping as it goes through the list. it only uploaded something like 50 images (from the last week since I did the export). All of this seems fine (if "unique" means things that don't need to be uploaded), except: every non-unique image (that's on the phone and in immich) is showing twice:
- Once with the indicator that it's local and not backed up
- Once with the indicator it's on the server, but no check (so not local)
It seems to be it's recognizing the duplicate images, but it's not merging them into one backed up image in the app.
I'm not sure if there's anything I can do here or if this is a bug. I thought one solution might have to be to delete all of the local copies and download all of them we want back on the phone. I tried that with a couple individual images, but downloading it didn't seem to change the icon to the cloud+checkmark so I didn't want to do it for everything.
Is this a bug? Known deficiency? Anything I can do to fix it?
Edit: to note - I've tried logging out/back in, clearing cache, completely uninstalling the app
53 Replies
So the phone shows both version, one on local and one is uploaded from your downloaded GPhotos?
What do you mean by Mattwilson method?
Do you know if Photos you downloaded from Gphotos is compressed or not?
Can you go to settings in Immich app and enable troubleshooting mode in the advanced section then go to a set of duplicated image and pull up the detail/info then take a screenshot or copy the content here?
correct
Sorry, referring to using the guide found here: https://github.com/immich-app/immich/discussions/1340
Utilizes the mattwilson1024/google-photos-exif tool
GitHub
gphotos import + albums migration · immich-app immich · Discussion ...
gphotos import + albums With this guide, you STILL have to add images that are in multiple albums manually (except for the first instance of that image, so hopefully this isn’t too many) can get al...
The file has been modified somehow so the content is not the same
Check the modified time
i do see that, but i dont know what causes that
looks like it's the ones i uploaded via the cli have the changed "updated" date
ohhh, i maybe have an idea
question: when running that script to maintain all the metadata from gphotos, does that keep that updated date?
after i ran this, i had moved all of those files into one folder to run the cli on them. would moving the files change that update date?
i'm not clear on the source of the "fileModifiedAt" vs "updatedAt" values
updatedAt is database level
fileModifiedAt is file level
Looks like the file is modified roughly 5 seconds after it is taken
ah, interesting. so this still seems like an issue/bug? i uploaded these via cli, looks like the updatedAt date correlates to when that was uploaded. but the updatedAt matches the fileModifiedAt for the local version, so there's no way for these to ever sync up if it depends on updatedAt
No, the file on your local device is modified
ohh you're saying the fileModifiedAt is 5 seconds off
between the two
Yes
that's..... how's that even possible?
I am unsure
which phone are you using?
pixel 7
Would you disable the location service somehow so that it might strip the exif of the file?
disable location on the phone?
Yes
My wild guess
ok i got that off, now what?
okay, more info: i scrolled back far enough to images before the Pixel 7 and they don't have the duplication problem
thinking maybe this has to do with how the pixel does all the post-processing on images?
It could be
How did the image get upload to Gphotos?
here's an example of one of the JSON's. looking at that photo in app, i'm not sure if this would be on the google-photos-exif tool or not, but it seems the one uploaded to the server via the CLI uses the photoTakenTime for both fileCreatedAt and fileModifiedAt
Gphotos was the auto backup on the phone until trying to switch over to Immich
I was looking in the code for google-photos-exif. looks like this is the default. i do see the google metadata typescript interface maps the creationTime value but doesn't use it anywhere
it seems that tool hasn't been updated in quite a while. i could probably make a fork and make a custom version, but do you know any way i could fix what i have uploaded? otherwise i'm going to have to figure out how to empty all images from immich and redo it with my fork
The best way is probably to delete them from either the mobile app or from web
i had tried that, thinking that would fix things since it's working fine on new items.. but i tried on a couple pictures. deleted them, tried to use the download button (cloud with arrow) when looking at it. but it never seemed to show as having a local version also (cloud with checkmark)
So if it is uploaded and on local, it would shown as cloud with checkmatk
Cloud alone meaning on remote but not on local
right. so when i deleted the local one (cloud with line through it), i tried to download the cloud one to get the cloud with checkmark, but it wouldn't work. it just stayed empty cloud
i got the "Download Succeeded" popover, though
Hmm interesing, maybe a bug for this scenario
that's what i was wondering
I will take a look at this scenario later
would this also be a bug: the two items seem to be treated as identical (i'm assuming because of the name?) so it doesn't try to backup that version (it stays local, no duplicates in the server) but then the UI treats them as different because of the modify date
i'm assuming we'd want that to be consistent
I think this is expected
So if the file is remote only and then you try to download to local it would show the new file as local with the cloud crossing
Because the modifiedDate is different
but then you can't backup that new one, but it's not really clear why
i understand i'd probably have to rename the file
Hmm 🤔
I haven’t really test this scenario throughly yet so we might not considered this case
understandable
i'm a software dev, i get it for sure
in my case i'd love if i could just "merge" the two together somehow. but that can be a bit weird for sure
I will report this case to the team
thanks
Thank you for testing
of course
The local sync mechanism is quite complicated so we haven’t gotten everything down yet 🥲
definitely get that. you have to use something to say they're the same 🙂
if i wanted to, is there a way to bulk remove images from the immich side?
By selecting the date group from the web or the mobile app on the ui level or play with the database
😅
i can do database, i wasn't sure if there was a way to connect to it
Just temporary expose port 5432 on the database container and you can connect to it
Just remember to shutdown the port afterward
okay. and removing the entry from the db doesn't remove it from the file system, right?
so i might need to delete some stuff that way too
Correct
FYI - I tested deleting the file locally and downloading it on a different phone without this duplication issue, and that also doesn't seem to work
both Android
Can you explain clearer which operation doesn’t work?
ya sorry, was going to edit that
i had an image that was local and backed up. i deleted it locally, so i got it to show the cloud only/only on server. i tried downloading it and it won't go back to cloud with checkmark even though i'm getting the Download Success message
ooohh more info: i found the images. they were downloaded to the Pictures folder, which is not part of my selected folders
One problem: after downloading an image, it gets a new localID (and new device id if previously uploaded via CLI) that's different from the one stored on the server, so the downloaded asset cannot be merged with the one on the server. It can also not be uploaded because it's hash is identical, so the server rejects it. To solve these cases we need Client-Side hashing etc... which will be worked on I guess at some point in the future.
For now, either backup assets from the mobile app (and not via CLI from Google takeout) or manually fiddle with the database entries and change their device id and possibly even deviceAssetId/localId... I would go down that route 😅
Oh wait so it's not currently expected that if you download an asset that's only on the server it changes over to a "cloud with check mark"? I guess I had assumed that's how that would work
I wanted to check, but it does appear that is how the ole GPhotos works, so that might be why I assumed. Sounds like maybe this is a direction that might already be planned?
Yeah this is the usecase we didn't really anticipated 😄
but it should work like your expectation
so this is a bug we would fix
🙂
Sounds good! Appreciate the info