Hashing numbers don’t match
Im opening a new issue here to keep it localized.
I installed both immich app and web anew fresh
No items have been synced yet, I’ve let the beta timeline hashing complete first
This is what I see now and it’s not matching the found items.
1. found items are wrong (but it’s been mentioned that this is actually expected as it counts all items including shared albums). I’ve ~9k images, it found ~14k.
2. hashed items are off by a few, yet it states to be finished with the job. The discrepancy matches exactly the count of items in „recently viewed“ of the iPhone photos app. Not sure if that’s coincidence.
3. it states „memories 0“, but that’s wrong.
Before I proceed with actual sync now I’d like to know what exactly is causing it and if it’ll cause issues (well, I’m sure it’ll: it will try to sync either number of items and thus either never finish sync or oversync, one of both)
All systems are up to date on latest version.
36 Replies
:wave: Hey @smileBeda,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:my gut feeling is again, related to Shared Albums icloud assets, those mismatches could be assets that are no longer accessible but are stil presented in the iCloud database 🤔
Is there any way I can find these assets it tried to hash, so I can check in iCloud?
This is a rather important part I think because if this is off, all other will be too I assume
You can export the sqlite database and send it to me. I can help look for them
The option to export the database is in the beta stats sync page
Sent in pm
Hey can you also send the latest mobile app logs?
Will do as soon i moved these out
1. The discrepancy between reported local files and hashed files finally is cleared: _I had unsupported files like
.mod and .mts in the photo library, which somehow where possible to add in past, no more now...
2. The new mobile app logs are sent in PM
Unfortunately, the import did not complete, and hash numbers are now again out of sync (after letting sync run overnight)
@Alex do you want me to open a new thread to go through this or proceed here?
Basically the behaviour was:
- yesterday hashing completed as we know and numbers of local assets > hashed assets where identical
- I selected the album to sync (Recents (All))
- It started syncing and in the "View Details" I could see how things progressed
- I let it run (open screen, active app, phone set to never turn off) overnight and in the morning:
- Hashed assets went down in numbers but are now increasing again slowly approaching the local assets number. I added NO new assets locally in this time
- Sync stopped, in the morning when I came in there was no item in the "view details" at all, and I closed the app, reopened, which did not bring back any items in "view details"... until suddenly it started syncing again. I have unfortunately no way of telling when it stopped, but it stopped early because after I started sync it went relatively quickly to about 1500 assets, and now it is on 1768 synced assets. So most of the night it idled. And when I look at the details now, I see tons of items... but none are actually progressing, they are all on 0--MB/s
- The "Enable backup" button thing said nothing when i woke up but now again says "queuing 99/7578"
THere was no special action on my end other than wait with the app open and suddenly that process started again
I think there are at least 2 or 2 main isues here:
- why does hashed numbers go down again after they where 1:1? - why does the sync hang up? How to "retrigger" it without having to wait an undefinte amount of time once it hangs up?
Can you share the screenshot again?
Just noticed the counts of "backed up photos and videos" is also inequal to "remote assets" and all of these are inequal to web-interface stats
first says 1768, second says 1957, third says 1775
I am going to scratch this again, this ain't working with large libraries.
I think there was a thing to do an initial sync, a script or so? Is that immich-go?
My hunch is this is related to iCloud shenanigan, will need more investigating
the web stats might have included live photos
or vice versa
In any case the queue had all files on 0mb, something clearly wrong
I scratched, not worth waiting for long time on this one
about that tool to do initial sync of large libs, is that something I dreamt up? or simply not the recommended way?
I will need to do more investigation, I have pretty large library but all my assets are locally on my phone
and it churns through very quickly
so I need to recreate a large iCloud library to figure out what exactly going on here
Thanks for helping with testing and reporting
I cant put all my photos onto my phone, it blows the memory
I do have them all on my mac thou, so that is why I am asking if there are other ways
@shenlong-tanwen Do you have any hunch on why the hashed assets number going down?
yeah the upload will download, then upload then discard the downloaded files from iCloud
Clearly with the phone this one wont do, be it because it has to pull from icloud a few assets, or my internet not good, or what it be...
I think if I sync the initial thing without the app, but I am not sure if that immich-go is the right way?
You can use icloudp to get all the assets from iCloud and then upload them to the app via the web/CLI or immich-go
iCloudPD?
yes
iCloudPD: ... it downloads from iCloud even if I already have them on the mac
Apple Photos export (wow, did not even know it exists!): ... useless. Crashes at a few pics
osxphotos (python lib): ... guess what, actually capable and reports... 14641 found items, exported 11445 items, missing 4828 (shared albums?). Elapsed time: 0:01:14
Analyzing those missing items, I can see them locally in my mac in the photos library folder and also in photos app. For some reason, they are labelled as missing thou and are not exported.
According GPT this is because it is a shared folder and thus while "visible" ... not in the main database (even if it is MY foto in MY shared album)
I think... Photos app is massively more buggy than anything else I ever saw.
Basically, you are either locked in or you ... well, better do not ever start using it becuase you may just not be able to actually "own" all that you put into there. Not surprised, but it is annoying.
I just remembered the weird thing were the updated time of the local assets were changed by iOS when we were just swiping through them in the Photos app. And I think we reset the checksum when the updated time is changed. Let me test this real quick. If that's the case, this would mean that the app is rehashing few assets every-time you open it after using the photos app, which could be a problem on the older hardware
I believe your device just amplifies the existing issues with our implementation, which is a good thing because we can finally see and try to address them. Hopefully, you should be able to use the Immich app to backup your library soon-ish 😅
Oh comon now...
After the osxphotos esport, I can see all pics in the folder when I open them in mac
However, on the immich web intereface, about 80% of those photos are shown like in the screenshot when clicking upload and selecting the folder where they are in
And uploading them does nothing
When the "file icons" actually show a "preview" then it uploads them
yet the photos are intact, thy are visible on my mac (see second screnshot)
🤣
I have a secret to share
you can actually download the Immich iOS app and run it on Mac
Right, I did that in past
then you can select the folder to upload photos from your Mac to Immich
damn
not a secret anymore
but that then duplicates all images, as soon as you install it on iphone
hmm
I guess photos on Mac is not the original version then
Not sure, but I asked in past and you said it is intended to NOT be installed on more than one device
As in, it was expected that to happen
we never fully tested it
so looks like it doesn't work
I did that in past, yes, and it basically duplicated everything. I asked here in discord and you suggested to use only one instance (there should be still a thread I think)
so Photos you have on Mac are not physically on the device then, probably still on iCloud
No, they are
Fully on my device - on mac
Not on iPhone, no place there
I see
alright scratch that idea
I have the data ready. Just that now https://discord.com/channels/979116623879368755/1407378678630973550/1407761671098994760 happens
lol
I am closing this, although there clearly is room to improve this... this aint a github ticket and i assume it wont be used as "workable task"
If need of course re-open
This thread has been closed. To re-open, use the button below.