Is there any plan to continue these PRs about race conditions on updating photos right after upload?
This issue is stopping me from uploading my library using immich-go. It was raised a while ago and here by @Daniel (sorry ping), and there were a couple PRs opened to work on it (1 and 2).
But both seem to have died. 😦
Is there any workaround or temp fix I could manually do with immich-go to overcome the limitation in the meantime? I'm unsure what causes the issue. Maybe a delay in between the upload and the update?
Any guidance is appreciated.
85 Replies
:wave: Hey @Chaos,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.
[Issue] Tags not attached to images (simulot/immich-go#990)
[Issue] [API] tagAssets doesn't tag all assets randomly (immich-app/immich#16747)
[Pull Request] fix: asset update race condition (immich-app/immich#16901)
[Pull Request] fix(server): Race condition in server jobs (immich-app/immich#17561)
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:Alo?
The workaround would be submitting all those asset updates after all uploads are fully processed. I don't think you can do that right now with immich-go
Long term this needs to be fixed on the immich server but it’s not a small job
thanks both for stepping in
Yep. I still believe my PR should've fixed it, but according to Simulot it hasn't and I didn't get around to looking into it again :/
But we are aware that this is a serious issue
is it until ALL uploads are processed? Or until all metadata extraction jobs have happened?
I believe the assetupdates already happen after each upload has completed
The latter. Until metadata extraction is done for all assets
yeah, I was fixated about this being a tags issue, but being metadata is a huge deal
:PeepoYES:
and even when the metadata job is paused during the upload, that doesn't help, as immich-go already tried that
Hmmm I would need to look into it deeper again but off the top of my head I'd expect pausing metadata extract until the updates are done to solve this too
hmmm ok. I'll try working on a patch that makes immich-go wait and check when the job is fully complete for all assets. And then carry on with all updates. No clue if that's doable for a large library though... but I'll try
*until the updates are done and the sidecar files have all been written, which is another job
I believe I tried doing it for the tags already.
1. Complete all uploads
2. Unpause metadata job
3. Update assets
And it didn't work
2025-08-03 14:43:59 INF Immich Job command sent pause=storageTemplateMigration
2025-08-03 14:43:59 INF Immich Job command sent pause=backgroundTask
2025-08-03 14:43:59 INF Immich Job command sent pause=metadataExtraction
2025-08-03 14:43:59 INF Immich Job command sent pause=videoConversion
2025-08-03 14:43:59 INF Immich Job command sent pause=migration
2025-08-03 14:43:59 INF Immich Job command sent pause=duplicateDetection
2025-08-03 14:43:59 INF Immich Job command sent pause=thumbnailGeneration
2025-08-03 14:43:59 INF Immich Job command sent pause=smartSearch
2025-08-03 14:43:59 INF Immich Job command sent pause=search
2025-08-03 14:43:59 INF Immich Job command sent pause=sidecar
2025-08-03 14:43:59 INF Immich Job command sent pause=library
2025-08-03 14:43:59 INF Immich Job command sent pause=backupDatabase
2025-08-03 14:43:59 INF Immich Job command sent pause=notifications
can you clarify which of these do I need to "unpause" and wait for?
metadataextraction for sure
sidecar?
Sidecar shouldn't matter in this case
What you probably want is
- stop both
- update
- wait for sidecar writes to complete
- metadata extraction
- sidecar update
(not a 100% guarantee)
sorry I don't understant that list.
Update I guess you mean upload.
Sidecar write happens separate from any job?
Update I guess you mean upload.Yeah sorry
can you clarify the last 3 steps in terms of jobs? Does sidecar writes start automatically even with paused jobs?
https://github.com/immich-app/immich/blob/main/server/src/services/asset.service.ts#L295 that's a dedicated job yes
any way to check via API?
and when you're saying stop both...? Metadata extraction and which other? Since you said the sidecar job shouldn't matter.
Actually SidecarWrite is in the sidecar queue too
So as long as you're not uploading sidecar (xmp) files you should be fine only pausing metadata extract
And from the API you'd then check that the sidecar queue is empty
1. Stop all jobs
2. Upload
3. Wait for sidecar write to complete (not sure if I can check)
4. Unpause metadataextraction and ???
5. Update all
(Immich-go role ends here)
6. Start all jobs again
7. Let sidecar updates to happen.
is this it? Can you correct or fix my mistakes please?
1. pause metadata extract
2. upload
3. update all
3. wait for sidecar queue to become empty
4. unpause metadata extract
ok so it has nothing to do with metadata extract. That was already paused from start to finish and didn't solve it. I thought we had to let it run before the updates.
You need to keep metadata extract paused until the sidecar queue is empty
You cannot resume it the second you've updated all assets
so, to check the sidecar queue, even if the sidecar job is paused, is it this same endpoint? getAllJobsStatus
But as mentioned here you don't want to pause the sidecar job
so even when the sidecar job is pause, the write will run on its own. Is that it? So just check for its queue
ahh ok
got it now
not pausing sidecar at any point. Just wait for it in between upload -> update, and then update -> start metaadata. This right?
Wait for it until the update is done too
There cannot be any metadata extract jobs running before the sidecar queue is empty
that's what I meant.
Upload
Wait check for sidecar to empty
Update
Wait check for sidecar to empty
Start Metadata job.
thanks for the guidance
will get to work
Again no 100% guarantee though :/
well, it's my only lead, might as well try :S
it's this, or not being able to migrate my library 😣
It didn't seem to work. 🙁 Log here
Any other ideas that I can try? Any thought is good, I'm happy to iterate and test any possible working leads.
I think I've got it.
So to summarize:
1. Stop all jobs except the sidecar job (which needs to stay running)
2. Allow uploads and updates to run normally in immich-go
3. Wait for the sidecar job queue to clear
4. Resume all jobs.
This did not work. There was still random assets not tagged or where metadata was lost.
I went and tried another more elaborate step.
1. Stop all jobs except the sidecar job (which needs to stay running)
2. Upload all assets first without applying metadata updates
3. Wait for the sidecar job queue to clear (ensuring all sidecar processing is complete)
4. Resume only the metadataExtraction job (to process the uploaded assets)
5. Wait for the metadataExtraction job queue to clear
6. Pause the metadataExtraction job again
7. Apply all metadata updates (tags, albums, descriptions, etc.) to the uploaded assets
8. Wait for the sidecar job queue to clear again after metadata updates
9. Resume the metadataExtraction job one final time
10. Wait for the metadataExtraction job queue to clear after metadata updates
11. Resume all remaining jobs and finish the process
THIS WORKED! 😭 🤩
3 test uploads so far and all the assets were perfect.
Awesome! I suppose you could merge 9-11 if you assume the user won't start uploads again right after, but this is probably also good :)
probably could, or maybe even simpler versions would work. But tbh, I was just trying to split into steps to get the best chances. I didn't want to take risks and get it working, even if not perfect efficiency. It's just a patch as proof of concept.
I'm sure the immich-go dev will be able to get the learning and polish it a ton
and maybe it gives you guys a clearer idea of how to fix it in immich too, which is the root cause at the end of the day
Yeah I think we already have a pretty clear idea of the root cause but this is definitely helpful just to confirm!
Please do keep us updated if you should run into issues with this again or if anything changes
So do i understand that right: this might happen in my workflow: uploading through immich-go directly in albums (storage template active)?
AFAIK yes, any upload via immich-go will currently encountrer this issue, and some assets will lose tags and/or metadata, randomly
Thx, although that‘s not very nice. Going to need to check metadata now of every image i already uploaded.
it's pretty easy to see for the ones with missing metadata. The ones where it failed, they lack an original date so they end up with the modified date, so they get dated for the day of the upload.
The ones with missing tags, it's harder to see though.
Thank you for that information. I set some exif data via script and checked some exif data via script, otherwise uploading would have aborted. i also have a exact copy of all files before upload.
so at the moment i think about writing a script which checks all immich elements and compares metadata with metadata of my files and reuploades them as many time it is needed until metadata is how it is supposed to be.
lol.
but have to check api first, if it allowes for that and think once again about it.
So basically, as of current state, immich-go is unsafe, in the sense, there's risk of loosing metadata
immich-cli would work, because it doesnt apply meta data explicitely from json, but extracts it directly from image exif
So, if i use a tool like https://github.com/Xentraxx/GooglePhotosTakeoutHelper
Which organizes takeout dump into albums and directly applies/inserts exif data to images from json
And thn i use immich-cli to upload those images/albums, it would work, since it wont depend on meta-data json files and doesnt need to seperately apply metadata
Is that correct understanding ?
as long as you don't apply any tags, it should
I don't know immich-cli that much, but AFAIK it only has an upload command. So there shouldn't be any "update" API calls.
so that feels like it should work
right, all exif data will already be in images and using immich cli will send no "update" api calls
so it would work, except tags, as cli doesnt support tags
plus, i can preserve albums
as the tool i linked can create sub dirs for albums
and immich cli can create albums from subdirs
I will post the findings on the immich-go issue, so ppl are aware of alternative solution available
I'll try it first
I also have the patched Immich-go mostly working. I've just spent the last few days doing large takeout uploads to find issues and solve them. Since every upload takes several hours, it takes a while
Can u share link of ur patched fork ?
I assume, what you fixed is
- wait till sidecar and metadata extraction job queues are done, so now there are already sidecar and metadata extracted (so they dont get overwritten)
- And thn u apply metadata updates from google's json
The other potential fix on immich-go side can be
Its possible to send a "sidecarFile" in multipart form data, when creating assets through api ("POST")
so, instead of "updating metata data from google's json - (by a update api call)" - if immich-go itself prepares a sidecar file for immich
and creates new asset, passing along the sidecar file, it would be used by immich and not get overwritten when sidecar/metadata extraction jobs runs
this could be much simpler thn waiting for sidecar and metada extraction queues to be cleared
https://github.com/Chaoscontrol/immich-go/tree/usable-merge
the racing-issue branch is the one exclusively with the fixes. The usable-merge includes that and it's the one I use with some extra features
Ah, ok, yes, i checked tht fork, but wasnt able to find the specific changes u made...
so your fork relias on the logic i mentioned above, right..."wait for sidecar and metadat jobs to finish" before updating metadata
yep, exactly the process I outlined above
it's in the readme as well
Check this
I will try if i can get a PR with this logic
still testing though
Its possible to send a "sidecarFile" in multipart form data, when creating assetsAnd they wont get overwritten
it could work, but tbh I'm not that knowledgable myself. I just wanted a functional solution for myself. I'm doing it all via AI, so I don't think the code is good enough for PRs. Maybe just good enough to give the proper dev an idea of how to fix it.
Got it
This is a serious issue, and immich-go should alert/warn users on its main repo page
Abt the potential risk of loosing metadata
Hey @Daniel , let me pick your brains for a sec.
My current immich-go process being:
1. Asset uploads
2. Wait sidecar queue
3. Unpause and wait metadataext queue, then pause
4. Asset metadata updates
5. Wait sidecar queue
6. Unpause and wait metadataext queue
7. Resume other jobs and complete
I wanted to ask for a few points after a huge run with 60k uploads (160k takeout).
- #2 is always empty. Is this normal? Please help me understand why.
- While waiting for sidecar queue on #5, I've noticed sometimes the queue gets stuck or super slowed. I'm normally running 15 concurrency, and it barely loads CPU or RAM. To get it unstuck I tried tweaking the concurrency midway, not sure if it helped, but it eventually picked up again and completed. The "speed" went up and down a few times, but really not connected with CPU/RAM bottlenecks. Can you think any reason why? Never seen this before in any other runs.
When this happened I had these logs:
- On #6, I got these constant errors (I think for every single file):
Any idea why?
#2 is always empty. Is this normal? Please help me understand why.You don't upload sidecar files (what Sudhir was suggesting) so that's to be expected Usually the limiting factor is IO (i.e., your drive) and not CPU/RAM. A concurrency of 15 may even be too high) (especially on HDDs)
yeah, it's an HDD. Ok, good to know. Do you think 10 would work?
USB3
I probably wouldn't even diverge from the default tbh
Maybeeee 7 or something
it usually worked at 15 in smaller runs, so that's why I tried higher
ok, will leave it at 5, no prob
HDD over USB isn't like the fastest (especially in terms of latency) combination you can imagine lol
Could this be related to IO issues from a too high concurrency? If you can verify that's not the case I'd bring in someone else
for metadata extraction I was using default 5, as I had also tried higher but it caused issues
Ah hm I've never seen those before
when this happened, I think I even tried lowering to 3 to see if it helped, and it didn't
@mertalev any idea what is going on here?
also, somehow it got fully stuck with a single 1 active job, 0 waiting, forever
at the end, after it processed the 60k files
not sure if related or diff issue
Still, for the issue at #6, is it fixable? What's exactly happening there and what happens if interrupted and other jobs resume without it completing? Will that still cause the racing issue? Or sidecar job being run it's already no problem?
I'll go for another attempt just with default concurrency. Bugger takes 12h per run now :Sadge: I thought it had finally worked all the way through
i think this is related to how the server library communicates with exiftool. it pipes things over to exiftool and waits a certain time to be confident the data is piped before adding more stuff, probably has some timeouts in place to detect if there’s a zombie exiftool process, etc. you might have choked the server with that concurrency to the point where the exiftool process cluster fizzled out
Hmm OK, thanks. I didn't understand all that, but I get that it might have boiled down to the concurrency anyway. We'll see how this next run goes with default settings. Uploading atm, will report back in a few hours.
the eli5 is that it expects certain things to be done quickly, but then the high concurrency makes them take too long
This time the run completed fine.
- I don't know if the #5 (sidecar queue) caused errors, because the logs have rotated from the HA addon and I'm remotely, so I don't have access to the host to check.
- #6 (metadata queue) still caused the same type of errors with default concurrency (5) for every single file. Not sure if they're the same as I didn't paste the last bit last time (the error reading exif data) and I don't have them anymore. But they look the same.
- The resulting files in the server have no tags, and all have date from today, so all metadata was lost.
- These were not happening in smaller runs (2k assets vs 60k now)
Thoughts @mertalev ?
Just completed another run with 10k assets and not a single error. :S Same settings. Not sure why it happens consistently when using a larger upload
I've started now another run with 2 takeout files instead of 1. My last idea of what triggers the error is if it's related to multiple zips being used. If not that, I really don't know why. @Daniel I could use any idea.
Sorry this is outside my expertise 😅
That's really something Mert would need to analyze and I have no idea in which rabbit hole he's in right now lol
Just for my understanding, once the metadata has been updated, and sidecar job has completed, what does the metadataExtraction job do?
Is this something that could be "delayed" and then resumed from the Immich server itself instead of during the immich-go run?
If that job doesn't complete, what's the state the assets are left in? AFAIK all the info (asset+metadata) are in the server already. Would rerun the job extract it if it wasn't done before?
Just trying to understand if I can just fix that rerunning the job or I need to delete everything and reupload from scratch.
It'll read the metadata back from the written sidecar file and update the tables (if there's a difference). There generally shouldn't be a difference but it's a nice safe guard to make sure the data in the DB is actually in sync with what's on the file system
so if the job has failed (ie the exiftool overloaded and crashed), the metadata is still in the server in the sidecars. So just rerunning it as many times as needed should eventually get the missing files in sync. Right?
Assuming exiftool didn't die while writing the sidecars, yes
the sidecar files are never deleted, and are kept in sync when metadata edits happen?
Correct. Metadata edits in the UI are written back to an existing xmp file if present, otherwise a new one will be created
perfect, thanks. I'll run again with concurrency 2 as a last resort, but I think that's not the issue and the long queue (190k jobs) is. If it fails again, I'll just try rerunning the job a few times and hopefully it'll be able to fix all the ones that had errors during the run eventually.
Tbf an HDD via USB is like the worst storage setup you can have 😅
So if it struggles there that's kinda fair I think
I mean, AFAIK for long term storage it's safer than SSD, isn't it? I'm aware performance is def going to be much worse, but I was going for safety (and cheaper cost/TB) here. Although I guess the backup strategy could be assumed to cover that risk instead. I'll think about it and maybe move to an SSD. My server only has a free SATA port tho, not another m2. 🙁
I mean, AFAIK for long term storage it's safer than SSD, isn't it?Not really. HDDs just die after some time, flash storage dies after x amount of data that has been written (that's why you also specify flash storage lifetime in TBW, TerraBytes Written). If you buy a couple of drives at once, they're likely to die at a similar time regardless of which type it is, if they're all being hit the same. In that case flash storage is also better because reads are essentially free and don't impact a drive's health nearly as much as on an HDD, so if you need to recover a RAID and need to get data off of a drive that's likely to fail soon too, an SSD is also your safer bet All that being said, you shouldn't ever rely on a drive not dying. Every drive will die, that's what a solid backup strategy is for But yeah with this message I was talking about performance/latency
good point. I will look into an SSD then
Also, quality drives will typically die later, so don't buy cheap etc
oof that's a hard one for me 😂 I do buy cheap. Will rely on the backups for that
@Daniel Dropping the metadataExtraction concurrency to 2 WORKED!!!! 😭 :peepoAwesome: 😭 :peepoAwesome: 😭 :peepoAwesome: 😭 :peepoAwesome: 😭 :peepoAwesome: 😭 :peepoAwesome: After a run of 23 long hours. No errors, all albums are there, all photos are tagged, all have their metadata, and no errors at all!!!
Niceeee!