Transcode job gets killed by failed video conversion.
Hi,
I've recently added about 1800 videos to my instance of immich. I'm currently running immich within docker, see my docker-compose:
When starting the job, it seems to fail after about 5 minutes. By default the job displays
34 failed, when clicking on missing it goes up to somehwere in the 1600s and slowly goes down. (I assume it's checking all the videos if transcoding is necessary.) After some time it displays the following error message within my console:130 Replies
Also: I'm on the latest build of immich, being the .1 version pushed this weekend.
It looks like it fails on a particular transcode, but I wonder why it kill the whole job.
It seems to also happen when starting the job for clip.
What tells you it's killing the whole job? The failure is for just one asset, it should still continue with the rest
Because the job stops running right after this error.
I also know its failing since the .avi isn't transcoded. And it does not match any of the ids mentioned within the log.
It being
7bd56e83-943a-46ab-bc00-3adb087bbabaThe job for the failed asset would stop, but the queue for the rest should keep going
Now that I look at it, there are tons of errors being generated by the different jobs
I take back what I said, it indeed finishes the job @bo0tzz. I was too quick to judge, however I wonder why it generates those errors. At the same time it does not seem to transcode my
.avi recordings at all, leaving them un-transcoded.When starting the Generate Thumbnail job, it seems to do the following
When extracting metadata it does the following for many of my assets:
WIth regards to the encode clip job, this one gets killed straight away and seems to be totally borked.
These are basically my issues with the job's at the moment, hopefully my logs provide some detail and can be helpful.
I haven't looked deeply but it seems like the files are broken in some way
I'll have a look
the files in MetaExtractionProcessor are fine
Lots of information is provided for SAN&0020.JPG for example, this feels more like some bug than anything else:
type timestamp with time zone: "0NaN-NaN-NaNTNaN:NaN:NaN.NaN+NaN:NaN"Did the files come from Google photos?
nope
All photos are either ripped from an SD-card or mobile phones.
This one is actually from a SANYO digital camera
more than enough metadata to extract
What I don't understand is why it seems to error out on timestamp time zone
Such as this avi file, which is not being transcoded:
just providing as much info as I can :)
There pretty old file might not have the correct EXIF header thus you see some of the erros
But failing on one job and killing the whole thing is new to us
If you can track down this one /photos/library/0eecf5ad-50a8-4a0d-9bfc-10d807df6191/2021/2021-01-12/20190815_123906_01.mp4
and if it is something you can share that would be helpful
will have a look sec
Alright, so bit of an odd one if you ask me. The file seems to be corrupted. At least, for as far as I can see.
is it corrupted on the original file?
i.e from your sdcard
Well, this one is actually from my phone I believe.
But it's a bit odd, since the 2021-01-12 folder contains two files. One that shouldn't be there at all, because it's a video shot in 2019. When I look at the timeline it also isn't there.
probably it hasn't gone through the pipeline
so no thumbnail to show
perhaps yes
I will have to manually remove the mp4 killing the clip job, however I find it weird how that one got corrupted.
depends on your hardware, 1800 video might take a while to finish everything
Well the job says it isn't running
I encountered a few situation where the file on the phone are corrupted
and they could cause issue
interesting
Well in that case I will remove this file and see if the clip job runs
It was an FYI
Yes, thanks. It does however skip over .avi files it seems?
I can't confirm on top of my head if AVI is supported entirely
You can download a sample AVI file and upload on the web to see if it shows up
encode clip gives me the following after removing the before-mentioned mp4
after that it seems to repeatedly do this:
it does, and it also playbacks which makes it even more odd.
Is it possible to remove all the transcoded videos? And run the job fresh?
are the logs for the example avi file
so I'm beginning to think it tried transcoding it, failed and now there is a corrupt file somewhere in the transcoded library that doesn't want to playback.
hm big chance that might actually be the case:
When I download this mp4 file, it downloads it as a
.3gp
Checked other avi videos in my library, older ones all seem to be in their original state, while some have been transcoded. So to best test this I would say removing all transcoded media and re-running the job. But I don't know what to exactly deleteThese error happen when you try to play video on the web and it can be ignored
Have you tried on Chrome? I know there are some issue with Firefox playback
Well, it happens on Firefox and my Android devices. I have not testen another browser
Android generates weird artifacts
I would guess there is something wrong with the trans coding process for AVI
The thing is, it did correctly do the sample avi file.
maybe track down that AVI file of yours and retest it?
But yeah, something is going wrong. But I can seem to find it within the logs
How would I try it on 1 file?
The erroes at the beginning of the post aren't related to the avi files
there might have a few things going on with the batch you upload
so let's try figuring out one by one
Ah you mean manually upload it again?
Yes
Ah okay, I will test this tomorrow. I'll report back on this, curious what that will do
Alright @Alex, I downloaded the
.AVI and re-uploaded it. Before doing this I went over to edge and it played the .AVI file without issue, and on firefox it didn't playback, nor does it playback on my android device
This is what the console shows;
However, when navigating to the asset it becomes clear that I can still not playback it on firefox, and it is still in .AVI format, as my settings are set to transcode if not in desired format, it should've transcoded this video file right?When looking in the directory
encoded-videos I've searched up the UUID of the image and it came back with a playable .mp4 file.
Is the file something you can share with?
Yes I can, but in private DM if you don't mind.
Yes please
Can you zip it up?
oh sure thing one moment
I will also include the encoded-video
There are other
.AVI files with the exact same issue.
Oh btw, if there is specific things you need such as more examples. Let me know Alex, more than happy to provide :)
@Alex you managed to find something odd in the file? 😃I haven't looked at it yet'
Oh no problem!
Yeah diving in facial recognition feature at the moment 😛
will circle back to this error later
Oooh, you don't even know how excited I am for that hah
The one I find most curious is this one, which doesn't really seem to be related to any file. Well it was, but it still crashes
@Alex I just tried running the clip encoding job again, but it still fails on activation. Are you experiencing the same issue, and is there a way to quote on quote "reset it"?
You can bring down the stack, remove redis and microservices container then bring up the stack again
I'm beginning to have the feeling it might be due to the way I have it installed. I'm making use of unRAID, and I installed it by using the community app. It worked, so I didn't really look back at it.
Using a postgreSQL database and redis seperately.
That's a different image & deployment method so we can't really diagnose and help if it's causing issues
The supported method is https://immich.app/docs/install/unraid#docker-compose-method-official
Unraid | Immich
Immich can easily be installed and updated on Unraid via:
Well that is the thing, I don't know if it is related to that
as it is very specific
hm
I can generate a compose file pretty easily, and it doesn't look at all weird to be honest.
but, I do agree is it hard to troubleshoot this. I will do it the official way and report back. The only thing I would like to know beforehand is if I can keep my current configuration
e.g., Redis DB and postgreSQL
If you keep your postgres database and library volume, everything should still be there after you move
(but take backups to be sure)
I backup daily off-site so I'm good there.
Alright, I will report back on this with my findings. As this might also be beneficial for other unRAID users attempting the arguably "easier way" out.
Did I do this correctly if I have an existing database?
I actually also have a seperate redis container
Assuming you pointed to your db in the .env, yes
and you probably still want the model-cache and tsdata volumes
Meaning I would need to uncomment
volumes: and the two you mentioned
ah I see how it is structured.
Hadn't taken a look at the .env yet
Any specific reasoning why versions are set for typesense and redis within the .env?If I want it to work in the same was as I had it before, would I need to submit
UPLOAD_LOCATION=/mnt/user/media? https://i.imgur.com/R25hqk7.pngAt the same time immich used to store some things within the folder:
It is also using it's own network, which I preferably don't want as I want it to use my own docker network
so I can easily expose it with my reverse proxy
like, where does the appdata go
This is what it looked like before:
I could add
- "/mnt/user/appdata/immich:/config" but I don't know how how the different containers react to this.
And also add
What are you guys tips on this? I want to replicate my old setup where config was in /mnt/user/appdata/immich
and my uploads and other data such as encoded-videos, thumbs, library under /mnt/user/media
@bo0tzz Sorry to tag you, but would you be able to answer my questions? :)I don't see any versions set, which part do you mean?
Yes
This is stuff from the imagegenius container, you don't need it with our setup
You can add a container to multiple networks with docker compose, so you can add your reverse proxy's network to the immich_proxy container and link them that way
Immich's data is in the postgres database and the library folder. The volumes in the docker-compose (model-cache and tsdata) are automatically generated by docker, they don't need backing up
I was referring to this:
Because those are the versions we support & develop Immich against
It's good practice to always set a particular version, rather than just using
latest (which could really be anything)Okay, I am however making use of Postgres 15
That should be ok
Didn't encounter issues before so.
Thanks for this.
The main reason it's not 15 in our default compose is because we haven't updated and tested yet 😛
like this?
Sorry if I am asking dumb questions, not used to doing it this way :)
fair haha
Alright, thanks! That helps a lot, now it's basically down to getting the container in the same network as my swag reverse proxy. Which is in the network named "hypnosnet"
edited the config I sent, it should be that. Not sure if I also need to define the
immich_default network as well now.You might need to, I'm not sure, it's been quite a while since I've done anything like that in compose myself 😅
I will add it just to be sure
okay I managed to get it to work
I added:
to all containers
resulting in them all being in my own custom docker network
now I need to fix my reverse proxy lol
and that works as well
good
Alright, I can access the user interface and everything seems to be exactly the same (thank god lol). I will now test if things are still reporting the same issues
hm
When I try to open images, it does not load them. Big chance these images are still in cache. At the same time the clip job is also failing, but I'm beginning to get the feeling it might be path related?
Same for within immich-server logs:
bit lost on how to resolve this one though haha. I did get a lot further now, and the job doesn't error out straight away like it did before. This seems to be a mapping issue. Would you happen to know how this can be easily resolved? It is mostlikely something within database right?
Try running the generate thumbnails job first?
hm I could perhaps that helps
but it doesn't open the image itself either
I am wondering if it is still seeing the same path
as expected
I'm beginning to think there is a difference in path with the official docker compose and unoffcial image
The old compose had:
Whilst this one only has this:
When I upload new images, I am able to view them
So my presumption is that something isn't right within the database, however I don't know how to resolve this issue.
How old was your instance before this?
1 month?
but there is something underlying
with the paths of the files
Hmm, database should be OK then. Make sure the mount path of your library is correct etc
it got uploaded in the correct place, which is good.
I'm off to bed so can't help any further for now
No issue, I'm going to pause this for a bit as well. But it is odd to me how it completly seemed to have lost the other assets while the new upload goes in the exact same library.
being
media\library\0eecf5ad-50a8-4a0d-9bfc-10d807df6191\2023\2023-02-27 for this image
Thanks though, appreciate you taking the time to help me outas expected, the database has the wrong orginalPath set. https://i.imgur.com/klyuwbE.png I'm starting to wonder to what extent it is possible to reverse this. @Alex I'm tagging you as well, in the hopes you might've encountered this before. I am able to change the path, but my knowledge is limited when it comes to databases. I don't know to what extent I am able to mass edit this.
The old path seems to be
/photos/library/
The new path seems to be /upload/library
As can be seen be the export I did on the db:
Name of my test file is: f97nfb7wkska1.webpSo basically it is replace
photos with uploadI thought this might be the case, but if your instance is only a month old, it should be new enough that this behavior doesn't exist anymore 🤔
Basically: those paths are absolute, nowadays we store them as relative
interesting
The last line in my markdown was an upload I did just now
and the other ones are from the unofficial image
Meaning the first one is absolute and the second one relative?
to what extent is this fixable for me
Empty assets table?
I will continue to look in to this tomorrow. Interesting one to solve
You could try this https://www.postgresqltutorial.com/postgresql-string-functions/postgresql-replace/
admin
PostgreSQL Tutorial
PostgreSQL REPLACE
This tutorial introduces you to the PostgreSQL replace functions that search and replace a substring with a new substring in a string.
Replacing
"/photos/" with "" (empty string)
I think that should work but I haven't checked thoroughly
So be ready to restore a backup if it doesn't work :^)rofl
why empty and not
/upload/ ?
cuz it is relative?Because you want the relative path ideally
hm
The correct absolute path should work too, but relative is more robust
But I uploaded a new file via web and that is using
upload/library/0eecf5ad-50a8-4a0d-9bfc-10d807df6191/2023/2023-02-27/f97nfb7wkska1.webp which seems like an absolute pathNope that's relative
No leading /
ahh okay
well, it doesn't now either way so
UPDATE "assets" SET "originalPath" = REPLACE("originalPath", '/photos/', '');
that seems to have done it
now I need to repeat this for resizePath, webpPath and encodedVideoPath
good that worked
now the paths are fixed in postgres
@bo0tzz Are you sure about your earlier statement about it not needing
upload/?
I have changed one of my assets to contain upload/ which fixed the issue
yep, fixed it with:
all my data is available again
Alright, now on to the one thing I actually made this ticket for. The diferent jobs not working.A bit odd, both did not work when selecting
missing however, when selecting all it did work
+ I only have 18k images, so I have no clue what it is doing but I'm just going to let it run
bit confused as to why it is incrasing
I killed the jobs as all of them were going above my amount of assets, and there seemed to be n slowing down. I will test again tomorrow
Is this normal behavior? @bo0tzz
Tag objects will be double your asset count
But I have 20k assets
So it doesn't really add up
I will just run all and see
I suppose nothing can really go wrong
It's so odd, as soon as I hit generate thumbnails tag objects and clip encodes go off as well
But they go higher and higher in number waiting
That's expected
As thumbnail is needed for tag objects and clip encoding, so it would kick off the subsequent jobs as well
Ah sounds logical
It does seem to work now, so that unofficial image has some quirks especially database related
Got it all working now and it doesnt killing itself
This is going to take a while lol
All my findings are within the thread as I have documented everything
I wonder why playback is so bad on firefox
Codec issues I think, proprietary stuff
I wonder what other good browser I can use
and yes, most likely
Been using firefox for ages, but it's letting me down haha
If it's a 4k video originally it could be that the transcode for it isn't done yet
At least for me, playing a 4k original kills my browser lol
either Brave, Chrome or Chromium should work I believe
yeah currently viewing on edge without any issue really
there's basically just firefox or chrome(based)
edge is based on chrome as well
yes
I am however wondering why the coloring seems to be so off on some transcoded vids
HDR
nope
@Alex I just tested and it seems even on the official compose build, it does not encode the .avi files I mentioned before. They are in a unsupported format. When trying to playback on mobile the file keeps loading, within Chrome it does seem to play without issue. I don't see anything within logs on tablet. I will have to check the logs within the container. Just wanted to let you know, as that was one of the main points of thus thread. I will provide the logs later. :)
It could also well be th3 case that it does encode, but it is not displayed within the ui. As I remember finding an mp4 format under encoded. I will have a more in depth look in the database as well. That way we have enough info to go on.
I did some more digging. It seems that
.avi IS getting encoded properly, when looking at the video it also does not display .avi extention.
However, there are also .AVI videos, which are not getting transcoded anc throw a MIME error. These DO display .AVI when clicking on the I for information.Probably just missing a .lower() when checking support?
I would assume, because both are essentially the exact same.