not all photos showing after cli upload
I have tried a couple of times now to upload folders through the CLI and done so year by year, this works well until I get by the 5 or 6th year or so.
The images after that just don’t show. There are no jobs running, just the machine learning tagging. It’s around 8600 images so far. I automatically let the upload create an album per folder and also for the final uploads the albums look like broken link thumbnails.
52 Replies
That would seem to indicate there are no assets to display in the album. Either the picture type is invalid (skipped) or immich could not generate a thumbnail. Can you check the microservices logs?
And/or verify the file extension for pictures in that year.
No as when I start with the folder it shows up and then the final folder doesn’t show
I have 2005-2009 at the moment. 2005-2008 show or 2009-2006 show depending where I start uploading. I am currently doing it folder by folder and wait till even the tagging is done.
I'm having a hard time understanding what exactly is going wrong . Can you clarify what you mean?
Are the photos picked up by the CLI or excluded from the list?
Are they included in the list, just not uploaded?
Are the photos uploaded to immich and you just don't see them?
Are the photos uploaded to immich, they just don't show up in the album?
I have folder 2005-2009 ready to be uploaded. I upload them folder by folder and all goes fine. But then the photos from 2009 are not shown. Also the albums created show broken links.
The total number of images is around 8500 which is indeed the amount it should be. But I don’t see the latest uploaded images.
When I start the other way around (start with 2009 and then upload folder by folder until I reach 2005) the same issue happens
Except then 2005 photos are not shown
Can you check the admin > job panel to see if there is any thumbnail generation job is currently running?
No jobs running. Only the machine learning tagging
Thumbnails are 0 as all others
Try running thumbnails job for missing
Did. No difference
Can you check the microservice log when running the thumbnail job
Yeah, I rebuild everything and I’m in the process of doing it again.
my hunt is that those files might be too old and the thumbnail generation job might have issue generate the thumbnail
before running through the CLI
try manually upload those files on the web
just to exclude some conditions
But that doesn’t explain if I do it the other way around it is fine
What do you mean by do it the otherway around
If I start with 2005 then the final folder 2008 doesn’t show up. If I start with 2008 and work my way down 2005 doesn’t show
Order should not impact anything and it not something we can reproduce. If there is a runtime related issue you need to find logs about it
True. I was replying to the comment that they might be too old
That could still be true
I think it would be best to monitor the log to see if there is anything abnormal
How would that be if eg 2008 works fine if I start with this one. But not if it is the last folder being uploaded?
Yeah, I’ll dig into the log. And get back here with that
Just because your observed the same behavior did not mean it is caused by the same issue
One direction could be old photos, and the other direction could be something else like just missing thumbnails, unless you ran missing in both situations and can verify it did not work.
One thing to do would be to check the library folder and verify what files are uploaded on disk
They all show in the library folder
That would imply a thumbnail issue then
Yeah, so I’ll check the logs when I reach that point
Thank you for reporting and checking
My pleasure. I used photoprism but didn’t find it suitable and now moved to immich and already feel better about it. So thanks for this great piece of software!
Fyi there is at least one reported issue relating to thumbnails getting missed while uploading
https://github.com/immich-app/immich/issues/2115
GitHub
[BUG] Path not found errors during thumbnail creation in 1.52.0 · I...
The bug Hi, thanks again for this update with great new features! I noticed during upload of a bunch of images that a lot of errors along the lines of immich_microservices | [Nest] 1 - 03/29/2023, ...
It seems to fix itself by rerunning the missing thumbs job though. So that plus microservices logs should help us figure out what's happening.
[Nest] 1 - 04/10/2023, 2:05:37 PM ERROR [MediaService] Failed to generate thumbnail for asset: 9ec1e80f-aa0b-4233-aa79-17dfc02fa313/IMAGE
immich_microservices | Error: File not found - /usr/src/app/upload/upload/ff8b3f6b-35d9-4bb3-b538-4359cf989969/1fe3a619-b495-4f61-867c-b0a4b8d8778f.jpg
immich_microservices | at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29)
immich_microservices | at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)
immich_microservices | [Nest] 1 - 04/10/2023, 2:05:37 PM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=3710bf62-8202-40bc-ba21-fa41a8b27a66)
so yeah, it seems to be the same thing
but another weird ting I see is this:
immich_microservices | [Nest] 1 - 04/10/2023, 2:05:38 PM ERROR [MediaService] Failed to generate webp thumbnail for asset: 7dc6fbf2-e119-414d-b3ce-4f201789263d
immich_microservices | Error: upload/thumbs/ff8b3f6b-35d9-4bb3-b538-4359cf989969/7dc6fbf2-e119-414d-b3ce-4f201789263d.webp: unable to open for write
immich_microservices | unix error: No space left on device
even though there is plenty of space
but the uplaod folder is indeed empty, like the bug describes
When I recreate all thumbs I get this
[Nest] 1 - 04/10/2023, 2:25:21 PM ERROR [MediaService] Failed to generate thumbnail for asset: 0729ead7-b1aa-4c8f-8255-5ad8e1ffae64/IMAGE
Error: creating /usr/src/app/upload/thumbs/ff8b3f6b-35d9-4bb3-b538-4359cf989969/0729ead7-b1aa-4c8f-8255-5ad8e1ffae64.jpeg
at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29)
at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)
No space left on disk
Should double check the volume that's mounted to the containers. Something is out of space, which is causing thumbs to not be able to be generated. Explains why starting over and doing the opposite order works except for the last folder in both directions.
i checked the space and there is free space...
even from within the container
Can you post the screenshot for the space of the volume mounted in the container?
hmm
but at the same time I see this
Can you bring down the whole stack and bring it backup and run the job again?
sure
you mean do a regenerate all thubmails job?
generate missing one
same
🤔
The weird thing is when I recreate the stack and load this folder first it works perfectly fine
Then adding folders and at some point I hit the same issue
Could it be an inode issue? As the upload folder contains loads of files
What do you mean by loading the folder?
I used the API to upload existing folders 5 in total (2005 - 2009)
i started with 2005,2006,2007,2008 en then 2008 failed as per above
if I do 2008, 2007,2006,2005 then 2008 is generated just fine but 2006/2005 fails
so it seems almost it is related to the amount of photos 🤔
anyway, i deleted and rebuld the stack and pointed the upload to a different disk, se eif that makes a difference
I think I might have found the issue
Filesystem Inodes IUsed IFree IUse% Mounted on
tmpfs 2015839 1399 2014440 1% /run
/dev/sda2 15597568 320006 15277562 3% /
tmpfs 2015839 1 2015838 1% /dev/shm
tmpfs 2015839 4 2015835 1% /run/lock
/dev/sdb1 0 0 0 - /volume2
see the 0's vor volume2 where my upload was located...
i reformatetd the disk and now I get this:
/dev/sdb1 61046784 11 61046773 1% /volume2
How it suppsoed to be. It turned out the ssd was still formatetd as vfat and not as ext4 😦
gonna try again now and see what happens, because the 'no space left' made me think od an idnode issue and as such checking the disk
Sounds good, hopefully this will help prevent future issues you have running other applications
yeah, sorry to bther you guys, when it turned out to be my haste to plug in an ssd and forgot to (re)format it properly
No problem, happy to help
All is well now.. I have exceeded 9k photos already and counting!
So, the wrong fs format had some limitations on inodes or total file count?
Yes. It was vfat and showed up as 0 in df. The reformat (re)build the inode tables and now all is good.
https://helpdeskgeek.com/linux-tips/top-3-ways-to-fix-no-space-left-on-device-error-in-linux/amp/
Intersting, 👍
I’m up to 2012 already and counting 😅
Thanks for the quick replies and help