Immich CLI upload issues as it doesn't appear to upload everything
user@localhost:/mnt/pics/$ podman run --privileged -it -v "$(pwd)":/home/import:ro -e IMMICH_INSTANCE_URL=http://192.168.1.123:1234/api -e IMMICH_API_KEY=$KEYS ghcr.io/immich-app/immich-cli:latest upload --album --include-hidden --concurrency 5 --recursive /home/import
Crawling for assets...
Hashing files | ████████████████████████████████░░░░░░░░ | 78% | ETA: 20206s | 16303
Hashing files | ████████████████████████████████████████ | 100% | ETA: 0s | 206952/
Checking for duplicates | ████████████████████████████████████████ | 100% | ETA: 0s | 206952/206952 assets
Found 73766 new files and 133186 duplicates
Uploading assets | █████████████████████████████████████░░░ | 92% | ETA: NFs | 38.3 GB/38.3 G
Successfully uploaded 51943 new assets (35.3 GB)
Skipped 21823 duplicate assets (3 GB)
Creating albums | ████████████████████████████████████████ | 100% | ETA: 0s | 339/339 albums
Successfully created 339 new albums
Successfully updated 206952 assets
Adding assets to albums | ████████████████████████████████████████ | 100% | ETA: 0s | 206952/
My question is, in the first iteration of hashing it says 133186 duplicates were already found, but while uploading it additionally skipped for 21823 assets citing duplicate and resulting on only uploads of 51943 new assets from 73766 new files initially detected during hashing.
Could you kindly explain on what is happening and if this is normal? Also, from the reliability aspects of uploads from CLI tools, Is there some better way to verify all uploaded contents to get 100% assurance on uploaded assets?
Regarding hashes, is there someway to preserve in logs or files too?
Immich cli version using 2.2.34 and IMMICH server version v1.22.1
5 Replies
:wave: Hey @khai,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:do you have duplicate images? Immich tries it's best not to store duplicate images within the same user
Also, another issue I noticed is this
During first iteration of uploads, cli did not uploaded all the assets and leftover 3 assets for some unknown reason. so this was the second run of cli to upload those remaining files
user@localhost:/mnt/pics2$ podman run --privileged -it -v "$(pwd)":/home/import:ro -e IMMICH_INSTANCE_URL=http://192.168.1.123:1234/api -e IMMICH_API_KEY=$KEYS ghcr.io/immich-app/immich-cli:latest upload --album --include-hidden --recursive /home/import
Crawling for assets...
Hashing files | ████████████████████████████████████████ | 100% | ETA: 0s | 4405/4405 assets
Checking for duplicates | █████████████
Found 5 new files and 4400 duplicates
Uploading assets | █████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 31% | ETA: 4h5m | 6.1 GB/19 GB
Successfully uploaded 2 new assets (6.1 GB)
Failed to upload 3 assets:
- /home/import/Summer/IMG_7006.MP4 - Error
- /home/import/Summer/IMG_6700.MP4 - Error
- /home/import/Summer/IMG_7005.MP4 - Error
Creating albums | ████████████████████████████████████████ | 100% | ETA: 0s | 0/0 albums
Successfully created 0 new albums
Successfully updated 4402 assets
Adding assets to albums | ████████████████████████████████████████ | 100% | ETA: 0s | 4402/4402 assets
In this third iteration, it uploaded 3 items that failed from second iteration of cli run
user@localhost:/mnt/pics2$ podman run --privileged -it -v "$(pwd)":/home/import:ro -e IMMICH_INSTANCE_URL=http://192.168.1.123:1234/api -e IMMICH_API_KEY=$KEYS ghcr.io/immich-app/immich-cli:latest upload --album --include-hidden --recursive /home/import
Crawling for assets...
Hashing files | ████████████████████████████████████████ | 100% | ETA: 0s | 4405/4405 assets
Checking for duplicates | ████████████████████████████████████████ | 100% | ETA: 0s | 4405/4405 assets
Found 3 new files and 4402 duplicates
Uploading assets | ████████████████████████████████████████ | 100% | ETA: 0s | 12.9/12.9 GB
Successfully uploaded 3 new asset (12.9 GB)
Creating albums | ████████████████████████████████████████ | 100% | ETA: 0s | 1/1 albums
Successfully created 1 new album
Successfully updated 4405 assets
Adding assets to albums | ████████████████████████████████████████ | 100% | ETA: 0s | 4405/4405 assets
this is the fourth iteration to verify if there was still anything leftover, but nothing showed up. There was no any network,server,disk related known issues. So, is there any known issues with large file size upload or restrictions via cli?
user@localhost:/mnt/pics2$ podman run --privileged -it -v "$(pwd)":/home/import:ro -e IMMICH_INSTANCE_URL=http://192.168.1.123:1234/api -e IMMICH_API_KEY=$KEYS ghcr.io/immich-app/immich-cli:latest upload --album --include-hidden --recursive /home/import
Crawling for assets...
Hashing files | ████████████████████████████████████████ | 100% | ETA: 0s | 4405/4405 assets
Checking for duplicates | ████████████████████████████████████████ | 100% | ETA: 0s | 4405/4405 assets
Found 0 new files and 4405 duplicates
All assets were already uploaded, nothing to do.
Creating albums | ████████████████████████████████████████ | 100% | ETA: 0s | 0/0 albums
Successfully created 0 new albums
Successfully updated 4405 assets
Adding assets to albums | ████████████████████████████████████████ | 100% | ETA: 0s | 4405/4405 assets
I am not entirely sure, probably there are some. So it would have been best if I could verify those. I am also wondering if there is some way to check out the reliability of these uploads too, as I am not being entirely confident over the uploads due to several issues I am facing atm.immich-go is an alternative cli upload method if you prefer
it's a community member's project, but it works pretty well for large cli imports
Thanks for the recommendation, I would probably need to check that out if there was indeed some limitations or known issues with IMMICH CLI tools.