Some photos are not backing up (iPhone)
Earlier this week I tried to autobackup my entire photo library from my iPhone. However, when backup finished, it turned out that some (a lot) of the photos are not backup up (display crossed cloud in Immich).
Going through related posts here and GitHub, it looks like there may be various potential reasons, so I decided to start narrowing the problem down.
I ended up:
- creating a brand new setup (nuked db and UPLOADS folder) to start from scratch.
- Crated a dedicated album in Photos lib to experiment on a smaller subset of photos
- added 27 images to it
- configured Immich to backup that specific album
However, when Immich started backing up, it displayed that only 24 images are going to be backed up. And indeed at the end of the backup, there were 3 images with a crossed cloud.
- I tried to repeat these steps (including wiping the entire setup) and it does seem like same images are missing every time
- device logs indicate that all 27 images are “upserted” (whatever that means). Other than that - no alarming error messages in logs
- checked microservices logs - couldnt find anything alarming there either
- checked “library” folder - those 3 files are missing there too
Based on this, it does seem like device actually sends 24images instead of 27.
What are the recommendations here for troubleshooting?
46 Replies
Do you perhaps use shared album feature in the Apple Photos app?
Are those missing photos are shared with you from another user in the Apple Photos app?
No, those are my photos and I couldnt find any conceptual differences with photos that work correctly
I even tried to export that batch to a local drive, delete them from Photos library, re-import to Photos lib from local drive and try repeat steps with Immich - same results
Are these photos visible on the web?
Do you use a direct connection to the server or some proxy like cloud flare?
They are not visible on the web. Connecting directly to the server
Okay. It seems our backup code has a bug somewhere. Because backup screen only shows 24 assets while the app loads all 27.
Just to make sure: there are no identical/duplicate images in these 27?
As I was double checking differences between working images and broken ones - I may have found an interesting fact. I dont know if its true for all images that dont work, but Im using “shared albums” feature in Google Photos app and suspiciously photos that failed to backup were put in one of the shared albums in Google Photos app. Again, those are my photos that I shared with others, not the other way around
Correct. All images are unique
Okay, this is interesting. As I don't have an iOS device, I can't try this... Maybe @Alex can try to reproduce this scenario?
We could also add more logging to the backup process
I would have helped with debugging, but Im an Android dev, so pretty far from setting up an iOS dev environment. One thing I can do - is to reroute mobile traffic through proxy to see what client actually tells to the server.
No worries, we'll add some additional debug logging to the app, that should already help
Good data point, thank you. I will try to reproduce this scenario
I am narrowing down image set even further. Now Im adding 7 images to the album, but only 4 are getting backed up. Sniffing on network traffic, I indeed can see only 4 upload requests that client sends. So something funky is going on on the mobile client
Another strange fact: no matter what I do, broken pictures are not getting sent to Immich server for upload. However, if I duplicate such photos in Photos library, duplicates are behaving correctly. I exported both to my laptop to check their hashes and md5 hashes are identical.
let's try to add more logging to the mobile app. If that doesn't help - I think Im at the point of setting up a development environment and just start debugging it locally as I can 100% repro it
And you said the photos that aren't sent were shared in Google Photos?
at least some of them. But definitely not all. Looking at failing photos, I tend to think that those photos were at some point shared via different means (Google Photos, airdrop, etc). However, I tried to share one of the photos that works fine (and I definitely didnt share it before) via airdrop and it was backed up just fine
@Alex , do you have codepointers for the uploading logic on the client side?
what do you mean by codepointers?
GitHub
immich/backup.service.dart at feat/archived-on-mobile · immich-app/...
Self-hosted photo and video backup solution directly from your mobile phone. - immich/backup.service.dart at feat/archived-on-mobile · immich-app/immich
yup, this one
@Alex , ok, got an app debugger up an running with my real device attached. So I should be able to repro it. Will take the app for a joyride later today/tomorrow 🙂 Let me know if there are specific parts of the upload pipeline you want me to check
Thanks!
Hmm.. When I try to debug the app - I can't repro the issue anymore. Also I reinstalled the app from the appstore (so the internal app state is completely wiped) and I can't repro the issue with release version either. Could it be that app got into some weird state that wiping out server-side state is not helping? I.e. in all my previous tests I would wipe Postgres db + remove everything from UPLOADS folder. However I did nothing to the app itself. What kind of data do we persists on the app side?
I.e. currently if I add 500 images to the album, Immich app properly displays 500 items to be backed up
The only remaining issue - is some of the images are actually failing to upload (upload request fails with HTTP 500). Looking at the server logs - this is what it spits out:
2023-04-16T16:39:06.780751809Z EntityNotFoundError: Could not find any entity of type "AssetEntity" matching: {
2023-04-16T16:39:06.780765868Z "where": {
2023-04-16T16:39:06.780770791Z "ownerId": "c0d634e3-af6e-4bbd-a49c-0c2daeb37ffc",
2023-04-16T16:39:06.780775424Z "checksum": {
2023-04-16T16:39:06.780779730Z "type": "Buffer",
2023-04-16T16:39:06.780789196Z "data": [
2023-04-16T16:39:06.780794210Z 110,
2023-04-16T16:39:06.780798589Z 116,
2023-04-16T16:39:06.780802771Z 118,
2023-04-16T16:39:06.780807626Z 224,
2023-04-16T16:39:06.780812144Z 23,
2023-04-16T16:39:06.780816306Z 227,
2023-04-16T16:39:06.780820713Z 157,
2023-04-16T16:39:06.780824920Z 24,
2023-04-16T16:39:06.780829077Z 169,
2023-04-16T16:39:06.780833452Z 71,
2023-04-16T16:39:06.780837750Z 115,
2023-04-16T16:39:06.780841949Z 226,
2023-04-16T16:39:06.780846088Z 35,
2023-04-16T16:39:06.780850284Z 19,
2023-04-16T16:39:06.780854630Z 76,
2023-04-16T16:39:06.780858988Z 238,
2023-04-16T16:39:06.780863405Z 248,
2023-04-16T16:39:06.780867617Z 7,
2023-04-16T16:39:06.780871969Z 228,
2023-04-16T16:39:06.780876313Z
2023-04-16T16:39:06.780880634Z ]
2023-04-16T16:39:06.780885077Z }
2023-04-16T16:39:06.780889419Z },
2023-04-16T16:39:06.780893642Z "relations": [
2023-04-16T16:39:06.780897981Z | 0=e 1=x 2=i 3=f 4=I 5=n 6=f 7=o
2023-04-16T16:39:06.780902533Z ]
2023-04-16T16:39:06.780906896Z }
2023-04-16T16:39:06.780911158Z at /usr/src/app/node_modules/typeorm/entity-manager/EntityManager.js:638:39
2023-04-16T16:39:06.780915596Z at runMicrotasks (<anonymous>)
2023-04-16T16:39:06.780920662Z at processTicksAndRejections (node:internal/process/task_queues:96:5)
2023-04-16T16:39:06.780925463Z at async AssetService.uploadFile (/usr/src/app/dist/apps/immich/apps/immich/src/api-v1/asset/asset.service.js:72:35)
2023-04-16T16:39:06.780929959Z at async AssetController.uploadFile (/usr/src/app/dist/apps/immich/apps/immich/src/api-v1/asset/asset.controller.js:62:29)
@Alex , attaching debugger to the server, it looks like that last error is related to the fact that some images in my Photos gallery are actually duplicates (i.e. I manually duplicated them - mostly for photo editing reasons). I.e. they have identical checksums. In this case upload fails (see this line: https://github.com/immich-app/immich/blob/feat/archived-on-mobile/server/apps/immich/src/api-v1/asset/asset.service.ts#L111). However the actual error is coming from a later code (which could be a bug) which tries to turn duplicate upload into success and retrieve original asset: https://github.com/immich-app/immich/blob/feat/archived-on-mobile/server/apps/immich/src/api-v1/asset/asset.service.ts#L126
GitHub
immich/asset.service.ts at feat/archived-on-mobile · immich-app/imm...
Self-hosted photo and video backup solution directly from your mobile phone. - immich/asset.service.ts at feat/archived-on-mobile · immich-app/immich
Thank you for the detail finding and report. i will take a look
looks like there is an open issue for it already: https://github.com/immich-app/immich/issues/2269
GitHub
[BUG] iOS App Repeatedly Uploading to Immich Server and Failing - E...
The bug When using the iOS app to upload images to Immich, I faced an issue where the app repeatedly attempted to upload while failing to do so. Earlier, I had successfully used the cli tool to upl...
+1 on this bug
I took a bunch of pictures in succession and a few didn’t get uploaded. I can’t see them on the web ui. I need to check if it’s on the server as mentioned above.
Can you take a screenshot of your Photos app settings? Do you have My Photo Stream enable?
Not sure about Photo Steam
it is in the Photos settings not iCloud setting
Thank you looks alright to me
from the timeline, you showed, there are multiple photos that look quite similar, did you just take a bunch of photos in the same spot?
Yes
But I have examples of some that aren’t like that
What is the problem are you seeing?
duplicated photos or photos not uploading?
Photos not uploading
Can you help pull-to-refresh on the homepage to see if it help with refresh the state?
And I would appreciate if you can reproduce with a different set of photos that doesn't look like one photo
Trying that now since you asked. Lol
Thank you
Ok - this happens to me a lot of times. I asked the question and now it works… I did right before posting, ran all the jobs on the server to make sure all data was there. So I’m guessing that might have helped
:BO_dtz:
Will do more tests and see. But will assume data has been reconciled on disk and the database
Correct
they will be put in a queue for smooth processing and report the status at a later time once it is processed succesfully
Going to try to reupload my google takeout via cli… going to see if this is any better 😁
Thank you!
careful that you will need to match metadata for Google Takeout first otherwise it will deemed as different photos
I think I’m ok with dups for a couple of months. Other data is all different
there are opensource tools that can help you merge the metadata back to your photos from Google Takeout
Please do share
I don't know a specific tool, you can search Google for such tool