Unable to import many pictures: "Error getting device's uploaded assets"
Trying to upload a folder containing ~220,000 files doesn't work, I'm assuming it times out behind the scenes while processing all of that. Is there any way to verify that's the case or increase the time limit? Or increase the verbosity of
immich upload -r?
The same picture folder, albeit maybe a few hundred pictures smaller, imported without issue a few months back with an earlier version of Immich.
Another server with identical configuration, also a fresh deployment, imported 12,000 files from the same node client without issue which makes me think this has to do with the amount of files
I can get the logs for any of these subcontainers, just let me know which would be most useful
Setup:
Kubernetes Helm container (TrueNAS Scale Official)
Version 1.67.2_1.0.5 (latest for TrueNAS) - Fresh install/deployment
7.2TB Free for Immich, 467GB Library folder attempting to import
Thanks!!
165 Replies
What is your immich cli version?
0.39.0
Hmm 0.39 should be compatible with 1.67.2
If you try again, would the same issue occur?
I've tried a few times, yeah
Restarted the container and cli vm as well
Strange. Once they release 1.68 or 1.69 can you update the CLi to 0.40.1 and try again? We fixed a few things in 0.40.1
Just saw there's an update available for 1.69.x, I hadn't checked in a few days
I'll install that soon
Well same issue, now on 0.40.1 and 1.69.0_1.0.7
Same error
can you post the screenshot please?
Let me add a message to print out the error and publish it
should be up in 5 mins
it is published
can you help install the new version 0.40.2 and try again
it should print out the detail error message
Okay updated, waiting on it to index local assets first
I had in it tmux so I can scroll around but can't copy the entire output since it's in a vm(novnc). This seems to be the important bit
I can screenshot the rest in pieces if that's helpful
yes please
how many assets are you having on your instance currently?
2
really?
hmm
I thought it might have been millions to cause such issue like this
Where are you currently running the CLI on?
These are some weird error
CLI is on endeavour os (Linux 6.3.1-arch2-1) importing about 200k assets
I suspect that it might have been issue with the TrueNas
Can you spin a test instance on your endeavour OS using docker-compose method?
then try to run this upload to that instance to see if it works, meaning getting through the prompt
I'll follow the guide and try it, sure
My vm was out of date when I went to install docker-compose, so I updated everything (which also reinstalled npm and immich-cli) and retried the import to truenas just to be sure nothing changed. It was the same error
As for the test instance, I keep seeing "no space left on device". There was only 3gb left on the vm so I tried it on my laptop with >100gb free. Same error so the message is a bit misleading. Must be a docker issue, I'm looking more into it before I can continue
I was in ssh without realizing it - the instance came up fine on my laptop. VM imported test 2 test assets fine and it's running the 200k files now
Same error I believe, truenas has been ruled out
If you choose a subfolder , would that work?
Probably yeah, I tried
*.* before to just import loose files and it worked, got to the next prompt
I'll try a folder thoughPinging other @Contributor, so everyone aware of this issue
Tried a 62gb subfolder and it worked as well
Are you trying to do
*.* in a folder with 262k assets?
Ignore that, apparently I can't read 😅
When you get EPIPE it usually means that the other end closed the connection
Can you see any error in the server log?To be clear how big is the list of uploaded assets at this point? It is basically 0 right?
It is running into an error getting the "assets by device id", right @Alex ?
The error seems to occur before anything is uploaded, that's why I think the server log might be more interesting here
Which of these logs should I pull?
Yes basically , it's only 2
Got the log of immich-7b76bc597-bkvp5 from this command template
k3s kubectl logs --namespace "ix-$app_name" -f "$pod" -c "$container
Is there a way to increase verbosity? Or am I looking at the wrong fileThis is strange. Looks normal.
Do you have a firewall that could interrupt connections?
My router is fairly basic so I doubt it would do that but I'll check
Your router shouldn't be involved here at all right?
Given it's all on LAN side I don't think it would be
I can turn off these things and give it a test if that would make a difference? (TP-Link ER605)
Not sure though, it would have no reason to block immich transfers since it's all within LAN, and even if for some reason the firewall rules applied it wouldn't be considered ping of death or winnuke (icmp / port searching / urgent packet)
I realize though you could've meant software firewall, those I'm less familiar with. I'll see if truenas or endeavour came with those. UFW, IP Tables, and Apparmor are the only ones to look for, right?
VM has iptables, but no ufw or apparmor. Here's the iptables rules
Truenas has a long list of iptables rules and a few apparmor profiles(?), but no ufw. Here's those
I just wanted to check whether you had explicitly configured something. I don't think any default setups should interfere with that.
Okay
Then no I haven't messed with it
The thing that's is strange is the login endpoint works but getting a list of previously uploaded assets results in that write pipe error.
Obviously he can connect to the server, since it logs in. Also, the request is not an upload, but another API request to the server, so I think any upload related issues aren't in play here either. Can you access that endpoint in the browser after logging in so you have a cookie?
The url in the middle of this picture (/api/asset/UUID)
Is there mabe some form of concurrency going on?
What do you mean?
This works in the browser
Afaict that request in the CLI is running into an error
https://github.com/immich-app/CLI/blob/main/bin/index.ts#L188
GitHub
CLI/bin/index.ts at main · immich-app/CLI
CLI utilities for Immich to help with upload images and videos from a location on a desktop machine or a server to the Immich's server - immich-app/CLI
Is the cli preparing its api GET request before it's done indexing local files, then the request "expires" by the time indexing is complete? Since the number of files defines how long that step takes, timing must have an impact, as a smaller(faster) folder works fine
This is only an issue for large folders?
Do you have the option to try the new, prerelease cli?
Yes, I tested importing the whole photos folder (~220k assets) vs a subfolder (~38k assets) earlier in the thread and only the latter worked.
There was also an identically configured server/instance that imported a folder of ~12k assets fine from the same cli
Probably yes, is this installed through npm or built from source?
Try cloning the source code and then reading the readme in the cli folder
It should be pretty straightforward. Please let me know of any issues, we're aiming to get it stable after I'm back from summer vacation
Fyi We have a partially rewritten CLI in the main immich-app/immich repository, not the immich-app/CLI repo.
That's what @etnoy is referring to
Yes, should mention that you clone the main immich repo
Oh I see, will do
It is supposed to be done with indexing before fetching remote asset data. But something must be causing an issue if the same code works with a smaller folder.
Maybe the drive is busy or no more file handles are available or something weird like that
It's a very interesting thing, I'm intrigued
Worst case you could just import sub folders one at a time I assume.
I was thinking the same thing, but also intrigued on what's causing this, hopefully we can figure it out
Error with ts-node, am I missing a package?
Ah, do an npm install in the cli folder
What version of node are you using? node -v
installed yesterday so latest afaik but I'll check
9.8.1
Yeah, go to 18 instead
That one's ancient
There was a 9.8.1 version?
Try using nvm
Wait maybe that's npm
Oh mb, yeah that was the npm version
node is v20.4.0
Lol niiiiice
That makes more sense indeed 😅
npm i in the CLI directory should fix the errorsIt did
Going through the rest of readme now
Running import🤞
Error :( Something less vague this time though
Is it that file "000_0784.jpg" causing an error? Or is that a coincidence
I'll start a new import without that file and this time tee the output to a .txt
Seems like a coincidence though as that's the first file alphabetically
That seems to be the first file being uploaded
What is the full cli command you're using?
I don't think that immich is able to access that file in the container
"File does not exist"
How's your docker-compose?
ts-node cli/src import -r /mnt/dataset/Photos
It's a helm chart I don't think there is a docker-compose
I'll pull up the config thoughYou need to add the /mnt/dataset/Photos as a volume to the server and microservices containers
Just to double check - you're aware of the distinction between upload and import?
My mistake -
just checked the readme for new CLI and I went through too fast, I'll retry this with upload
I made sure my old cli command used
upload so that was correct at least
will post log once it's completeDid the upload get started?
It's still sitting on this screen, hasn't gotten to the percentage counter yet
Ok, keep us posted
btw, what happens if you run find /mnt/dataset/Photos? Does that command complete?
&>/dev/null
True
The progress bar should appear basically after crawling the local files
Why does it take so long though?
There's a lot of files
But just reading their names should only take some milliseconds, even for thousands of files
Previous screenshot shows 500 gigabytes of files
If I don't misremember, it also has to stat the files for basic file info
no
readdir
muuuuch faster
IIRC
I'd have to check my source code for when it stats the files
Should be a single syscall to list an entire directory including name and type
Pretty sure it stats every file when crawling https://github.com/immich-app/immich/blob/main/cli/src/services/crawl.service.ts#L14
Good news?
Nice!
Ah yeah that's probably the bottleneck
Taking notes about the initial crawl potentially needing a progress bar
I love that I put an ETA estimate + size indicator in the new cli 😄
Would switching to fs.readdir be a possiblity for you?
I don't have the code in front of me, but don't we need to stat every file in the crawl?
readdir returns that information without needing to stat
Ah, didn't know that
that's why find is much faster
I'll double check to see what info is actually read
not a priority anyhow^^
Thanks for the help !! It's settled at 5hr now, not so bad for a gbit link
will the old cli be retired at some point, so we don't need to worry abut debugging what happened before?
The old cli uses an old way of detecting duplicates
That we're discontinuing
so it will be retired as soon as the new cli is stable enough
Okay
Left it for a while and when I checked it stopped on this error. File is readable with ls.
I restarted the upload to try again since it seems random for the cli to stop on a file for no particular reason
Are you running this on the same machine that's hosting immich?
No, I have no way of putting npm on that machine at the moment
That's fine. I was just wondering if the jobs in were all running (machine learning, thumbnail generation, exif extraction) and if it was impacting the availability of reading the files from disk.
Is it reading/writing the files on the same disk?
Network share?
Otherwise I'm not sure why it would temporarily be unavailable 50gb in
/mnt/dataset is a network share going to the machine running immich, yes. And it is running jobs I believe
If it's the network share causing issues I can reboot the immich machine to enable virtualization, spin up a vm to run npm and upload from there so it's all within the same box
The disks and cpu haven't been constantly busy so there was most likely bandwith to spare even with jobs going
Well. My guess is all the background jobs led to the error you got. My two cents. There was a read/write spike around 18:00. Did the CLI run for an hour and then die?
Not sure how long it was running when it failed, but I paused the jobs just in case they affect the next run
That's what I was about to suggest.
Did you try out immich before doing this big data load?
Yes I ran a super old version (before search, machine learning, etc) for a while before I switched helm chart providers and needed to reinstall + reupload
There's still a bunch of stuff to fix, especially for large libraries like yours. Too much to do and too little time to do it all 😁
It's an impressive feat to take on google photos when so many other projects have failed at it ! Even if imperfect Immich has been very helpful so far
Failed at the preceding file...
I'm going to try uploading from a different client, that is a weird error
Thank you for helping testing this. This is the first instance we see this error from the CLI
Of course!
Is there be any way to add verbosity/context to this error or is it filesystem related?
it is the most verbose we can print out for that error
Okay
I think
If you can spin up an Ubuntu/Debian VM and try, it would be a great baseline test
Sure, I need to set up qemu and install Debian so that may be a minute
No rush here Orbit 😛
We are here to help to get you upload/import everything to Immich
Same error on my laptop
VM is up, just need to login with api key and start the upload from there
interesting
same file
I wonder if the file is corrupted somehow
Maybe , I'll try opening it
It opens fine so not corrupted afaict, I could take out that entire folder (866NCMTO) if you think it'd make a difference
The first error with cli-v2 was on another file in there IMG_3037, not IMG_3036
I think we can definitely try that as welll
I checked IMG_3036 against another working image in a hex viewer and theres no obvious error like a blank spot or repeating sequence
I'll take out the folder and try again in VM
Thank you
The common denominator between all 3 clients is CIFS (smb mounted on linux), should I try uploading from Windows or is there another plan to debug
smb mount shouldn't cause any issue
Okay
Is there any way to test for the second case here? Or would there not be more than one thread accessing
Interesting
I've set the cli to fail on first error. In the future I'm planning to have it do, say, three retries with exponential backoff
However it's tricky when you get an underlying OS error like EAGAIN, there's not much we can do with that unless it would clear on a retry
Do you have the option to run the cli from another machine straight from disk, ie not mounted via cifs?
From my experience, I'd say the issue is caused by cifs for sure
I do have a local copy of the Photos folder I could try
I'll connect it and start the local upload from endeavour vm
It made it past 53.8gb (!) I won't see it complete until the morning but there's good news
Awesome. I guess you'd run in to random cifs errors all along the way so this is probably the way to go
I'd be curious to see if cifs was causing errors in the old cli version as well, i'll try that if the upload goes well
Yes, the old version wasn't clear on errors.. I'm sure it ran into the same issues. We were just able to expose the problem now
To test, why don't you run the old cli on that other machine? 😎
There are 3 (+2 virtual) machines at play now lol, you may have to specify
The local copy i'm importing from now is on a hard drive plugged in where only the endeavour vm can access - I can start an old-cli upload in a second terminal pane if it won't interfere
If you'd rather I use a different machine though I could move the drive once it's done tomorrow morning
Ah. Let's wait until this upload finishes and we can play with it later
If you want to speed it up and you have lan connection you can do --skip-hash
It will send every file and not do deduplication before upload
That will only happen server side
For initial imports that will save time
Already uploaded files will not be duplicated
Oh good to know, thanks
Walked past the server and heard a lack of crunching hdd activity, checked to find this error.
CLI considered
tif and tiff supported (https://github.com/immich-app/immich/blob/1c5926553a7fba7a6eaf24be63181a61e6b763d7/cli/src/cores/constants.ts#L12) but server only supports tiff? (https://github.com/immich-app/immich/blob/1c5926553a7fba7a6eaf24be63181a61e6b763d7/server/src/domain/asset/asset.service.spec.ts#L103) I may be misunderstandingSupported types aside, @etnoy we shouldn't stop the whole show on this kind of error right?
Just skip the asset and carry on
Previously, the cli was too forgiving of errors. I'm planning to have some hike of retry mechanism, but just skipping past errors is not really the way to go
In this case, the new cli uncovered an underlying issue that frankly could lead to data loss if not acted upon
Most core UNLV Unix tools will fall on first error
Would you like to create a GitHub issue about that? That's an issue on the server, not cli
Sure we shouldn't just skip past, but there's a meaningful distinction between things like the error above (unsupported filetype) and more fatal stuff like connection errors
We should shout loudly about things like this of course, but I don't think we need to stop the whole upload
We need to sit down and categorize the different error types we can encounter and handle them accordingly
In the meantime, fail fast is the safe approach
Missing .tiff might be a regression (bug)
Sure I'll make an issue for it
https://github.com/immich-app/immich/issues/3429
It completed all 500gb, 100% just now
Thanks everyone for the help!
Awesome!
Let's go!
Nope you can help us track all the other stuff that doesn't work with large libraries 🙂
Sure! If i run into anything else I'll let you guys know
Don't worry Jason, I have a large library at home to test with: 6.4T and 570k items 😉
Noice