IMMICH upload destroy photos, sometimes
Hy there. Iam facing the issue that photos uploaded via Android APP are sometimes destroyed. Initially I thought its a problem of the cam in the phone itself, but then I noticed it happens with two different phones. So I uploaded the originals manually and found they are fine.
Please obserev the attached screenshot. Above is the original, below the one uploaded with Immich APP. You can see the disturbance on the right side of the photo. The disturbance is not always vertical, sometimes it is horizontal as well. Around 80% of photos upload correctly.
Upload is done via Wifi, it happens with Huawai P20 and P40 Mobile.
Any idea about that ?
30 Replies
:wave: Hey @bbernhard,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJAChecklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like
fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.GitHub
immich-app immich · Discussions
Explore the GitHub Discussions forum for immich-app immich. Discuss code, ask questions & collaborate with the developer community.
FAQ | Immich
User
GitHub
Issues · immich-app/immich
High performance self-hosted photo and video management solution. - Issues · immich-app/immich
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:
Hello, do you mind zipping up the 2 files and send it here?
Hi, thx for takimg care. Please find the original and the defect one in attachment. I use external library (folder is at my synology) and storage template. Probably the issue happens when the photos are copied at the server ?
If you can also include the docker-compose and .env file that will be great
both fiels are already included in the .zip
Hello, the defect file has multiple sections of bytes that are zeroed out
The most likely cause of this is probably an issue with your storage on your server and bad sectors
The zeroed section is about 512-516 bytes by my counting, which would fit perfectly with one bad sector
If you are curious you can enter the database and check the SHA-1 checksum of the file and compare to the file on disk
my guess is it will match the "OK" file
If I got you right you suppose a hardware defect of my HDD ? So far the Synology do not report any issue. S.M.A.R.T. HDD test says everything fine. Just thinking how I could temporarly try out other drive without scrambling up everything too much.
That’s what I think. SMART is not perfect
Unless you are using something like ZFS to detect the failures when reading you can’t be 100% sure
But you could compare the checksum like I said to see if it initially uploaded properly
I'm not saying you're wrong, but most hard drives only use 512bytes as "virtual" sector size nowadays, usually they are actually 4kB with 512Bytes only used as an interface/emulation layer. Either way, could maybe be resolved by checking the sha1 before uploading on the client, and later comparing it to the uploaded file on the server, retrying if the hash doesn't match.
yeah, I'm aware
it's 99% likely that the upload was fine in this case, and the corruption was later on
so immich can't control this
(we already do compare the hash in that way)
if the hash is already compared after writing to disk then this really is weird.
I could be wrong but I'm pretty sure.
Also we use TCP, which should handle all this transparently
TCP's error correction isn't foolproof though. I guess either the hash isn't compared by immich after the upload, or is compared before it's written to disk. I don't understand how the file could be corrupted otherwise, assuming it hasn't moved (e.g. through defragmentation)
I highly doubt it successfully writes and reads it once from the disk and then later suddenly corrupts itself, that would be really weird.
uh, that sort of seems like what a failing disk would do? works sometimes then not others?
also, it's not trivial to force a full flush and re read of a file from the storage controller through docker abstraction
either way, this is not really immich's fault, if we issue a write to the disk it's totally reasonable to assume it was written properly if the syscall returns OK
anything past that is up to the host
otherwise we are basically designing a new filesystem within immich
i dunno, if it really only happens when uploaded via phone, and not when manually uploaded
and fails in 20% of cases via phone, then there is something wrong
yeah, with their system... lol
literally never heard anyone else having this
maybe, could also be the phones themselves sending garbage, but if you compare the hash, then really that shouldn't be possible
just weird they're saying there is a discrepancy between manual and phone upload. if it's the server hdd, then there shouldn't be a difference however you send the images
I don't think we re-read from disk but we do re calculate the hash: https://github.com/immich-app/immich/blob/8dab5d37980a115fbc14dbf3486c18a07722883d/web/src/lib/utils/file-uploader.ts#L143
I could be wrong, this is not my expertise
it looks to me the hash is only used for duplicate detection
not for comparison with the client upload for integrity
i could be wrong though
Do you run any data scrubbing or other FS maintenance automatically/ periodically?
It doesn’t compare a hash on the server with a hash from the client
Just the hash of the asset against the database
I have now changed the backup location to another disk. So far it seems that it works correct right now. Made about 50 dummy pics, no one was currupt. Iam curios that this is real a hardware defect, because the disk is running fine, even a extended SMART check which run overnight didnt report any defect sector or other issue. Could it be that a race condition is causeing the trouble ? Previously the backup folder pointed to a folder within the external library, which is on a shared drive of my Synology.
However, ill observe further what is going on and be careful with my Synology backup.
SMART relies on the hdd itself to detect and figure out if it get's corrupted. It's far more reliable to run data scrubbing in your synology and verify data consistency using the raid parity data (If you're using RAID)
That being said, I'm not so sure it's actually the hdd with the issues, it could also be the phone sending garbage or the network stack having some corruption.
Data Scrubbing | DSM - Synology Knowledge Center
Synology Knowledge Center offers comprehensive support, providing answers to frequently asked questions, troubleshooting steps, software tutorials, and all the technical documentation you may need.
No, it happens with two different phones we observed it with Huawai P20 and P40.
I don’t think it’s a race condition
I’ve never seen it before and switching disks fixed it