Assets lost during migration
Dear all
I am finally migrating to immich and I am facing an issue with missing assets.
The first attempts were more successful in terms of completeness, but I was facing issues with the timeline as copying photos (from Synology Photos folder) changed timestamps. Now, several thousand photos seem to disappear - this situation seems to be reproducable as I have already retried it.
What I did:
- Mount USB drive on NAS1
- Create a tar archive with tar -cf <Synology Photos folder> to an USP drive
- Unarchive with tar -xf
- Delete all @eaDir folders with thumbnails recursively
- Mount USB drive to NAS2
- Upload photos with the following command:
docker run -it -v "$(pwd)":/import:ro \
-e IMMICH_INSTANCE_URL=http://<host>:2283/api \ -e IMMICH_API_KEY=<key> \ ghcr.io/immich-app/immich-cli:latest upload . --recursiveCrawling for assets... Hashing files | ████████████████████████████████████████ | 100% | ETA: 0s | 11929/11929 assets Checking for duplicates | ████████████████████████████████████████ | 100% | ETA: 0s | 11929/11929 assets Found 11929 new files and 0 duplicates Uploading assets | ████████████████████████████████████████ | 99% | ETA: 1s | 31.2 GB/31.2 GB Successfully uploaded 11910 new assets (31.2 GB) Skipped 19 duplicate assets (6.6 MB) Why does the progress bar stay at 99%? In the Server statistics overview I now see: Photos: 8.725 (16 GiB) Videos: 676 (13 GiB) Usage: 29 GiB I expected to see 11910 assets in total, but apparently there are only 9401. Is this a known error or am I doing something wrong? My wife has even 4 times number of photos. Thank you Patric
31 Replies
:wave: Hey @boku,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.I did not do no 5 & 7 as they are not relevant here, I assume. I am running immich on Synology DSM 7.2 latest release with portainer. and IMMICH_VERSION=v1.132.3. Caddy plugin in OPNsense
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:
could you confirm, do you have 11910 files in that folder?
you may want to give a try to https://github.com/simulot/immich-go
sorry, I do not know why it always creates an attachment
can you execute this? the server may be still processing and is not showing the full stats
oh, docker exec immich-db psql --dbname=immich --username=immich -c "select count(*) from assets where status = 'active';"
count
-------
11910
(1 row)
so it match
go to /admin/jobs-status in your server and check if there's any job running
give it some time to complete
right now, several jobs are running like thumbnails generation, search, facial recognition. But they are all from the latest immich-go run 1 h ago. But yesterday, all these jobs were finished and the number didn't increase
ok, let's wait for them to finish. the assets are stored in the DB, you can check the number of files under upload and library and compare if they match.
once all the jobs are done, if the stats are still wrong (assuming fs comparison is good), may be a bug on how it's calculated
thank you @NoMachine I will share an update. Is there anything regarding "assuming fs comparison is good" I can do?
count the number of assets in upload and library (if you arr using storage template), they should be 11910 if everything has been uploaded successfully
All jobs have finished.
In the server statistics overview, I see
photos: 8.725 (16 GiB)
videos: 188 (13 GiB)
total: 29 GiB which is less than 31,2 GB
If I press CRTL+A in the Photo web ui, I see 8913 pictures
I use default settings for storage templates:
- engine: no
- hash: yes
what's the file count in /volume1/docker/immich/upload/upload
you are the only user using immich, right?
yes, I am the only user now. /volume1/docker/immich/upload/upload# find . -type f | wc -l
12035
Do live photos count as 1 or 2 assets? Perhaps this is the difference?
they count as 2 in the fs, there could be some sidecar files too
Right, but how would that count within immich? I'm not at an instance to test, but it could be a possible place for the number mismatch
🤷
(for example, if you upload a live photo, does video and photo count increase, or just 1?)
I don't have live photos, so I can't test 😂
Or if there's a way to search for live photos then op could see if that's the number difference
I used live photos in the past but I do not know how they are persisted. Which file name suffix is that? Is there always a non-live photo created as well with the same name and a different suffix? If that is the case, I could delete my user again and upload only these 2 files, if that helps.
Creating a new user and just uploading one live photo would work too
I cannot find pairs of HEIC and MOV files. I do not know what the iPhone creates
I ran several migration tests. In PGAdmin "select count(*) from assets where type = 'VIDEO' and "ownerId" = 'ffe84ecf-b7a0-4c2c-87d0-e802158268ce';" returns 3185 but in the server-status page I see 188 videos. How is that possible? And the strange thing is, short after immich-go process I saw 2828 videos and then this number decreased - maybe it was even 3185 immediately after upload. Since you mentioned live photos, I used immich-go option "--manage-heic-jpeg=NoStack" (I hope my interpretation of this option was correct). The import logs do not show errors except warnings on "warning=can't read metadata: EOF". When I press CTRL+A in Photos search I only see 188. Why are all others hidden?
Hello NoMachine, I can see that cloumn visibility = hidden in table assts for 3185-188=2997 records. Do you have an idea why the upload sets them to 'hidden'? Can I simply SQL UPDATE this column to 'timeline'?
hidden is usually the video part of live photos, do you have many of them?
you should not update the DB manually
as I wrote, 2997 are hidden but they are for sure not all live photos
is up to you to investigate, since I have no way to know what those assets are
but what is the recommended way to unhide them if not via SQL UPDATE?
why would you want to unhide them? they are hidden for a reason, you have to analyze what assets are hidden and find out why. maybe they don't really exists in the FS. maybe they are part of a stack
yes, at least the ones I analyzed are pairs and have indeed the same name with a different suffix