Reprocess specific transcoded videos?

Hello found a album where all of the videos do not want to play in the web ui (something about mime support). It's odd as it's the same phone I've used for 1000s of other videos but this one day of videos does not work. On PC I tried 3 browsers and same Anyhow I see in we ui a way to progress the video again, awesome. However doing this did not change any results. I guess I was wondering 2 things. 1. If I find the matching file in encodedvideo folder and just delete it, will immich recreate it for me? (I guess if the answer is yes I am curious how to identify the name and path to the specific file in encodedvideo as well) 2. Similarly, I may want to just delete all the encodedvideo folder and start over. Would this work! My thought here is that deleting and starting over is more concrete to see the files were regenerated, vs trusting the reprocess thing actually doing something. I also separately have issues where that folder takes up space and never let go it's its space after deleting (separate issue I posted a couple of weeks backl Thanks
52 Replies
Immich
Immich3mo ago
:wave: Hey @RoachForLife, Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:. References - Container Logs: docker compose logs docs - Container Status: docker ps -a docs - Reverse Proxy: https://immich.app/docs/administration/reverse-proxy - Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA Checklist I have... 1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time). 2. :ballot_box_with_check: read applicable release notes. 3. :ballot_box_with_check: reviewed the FAQs for known issues. 4. :ballot_box_with_check: reviewed Github for known issues. 5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy). 6. :ballot_box_with_check: uploaded the relevant information (see below). 7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable (an item can be marked as "complete" by reacting with the appropriate number) Information In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider: - Your docker-compose.yml and .env files. - Logs from all the containers and their status (see above). - All the troubleshooting steps you've tried so far. - Any recent changes you've made to Immich or your system. - Details about your system (both software/OS and hardware). - Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h). - The version of the Immich server, mobile app, and other relevant pieces. - Any other information that you think might be relevant. Please paste files and logs with proper code formatting, and especially avoid blurry screenshots. Without the right information we can't work out what the problem is. Help us help you ;) If this ticket can be closed you can use the /close command, and re-open it later if needed.
RoachForLife
RoachForLifeOP3mo ago
Raw files on external library on the Nas, immich (dB and encodedvideo) sits on same Nas but ssd drive Using 1.35 immich
Immich
Immich3mo ago
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:
RoachForLife
RoachForLifeOP2mo ago
Curious if any of the mods could help on this one, thanks
schuhbacca
schuhbacca2mo ago
1) No I don't believe so. It will just error when attempting to play the transcode as it won't be able to find it. (You should not manually touch anything in the upload location) 2) To minimize touching the folder directly. You can turn the trnascode policy to "Don't transcode" then run the "Transcode All" Job. This should delete all items in the encoded-vidoes folder. Then you can double check and see if there's extra stuff in the folder
RoachForLife
RoachForLifeOP2mo ago
Ok thanks I will give #2 a shot later this week. Thanks
RoachForLife
RoachForLifeOP2mo ago
@schuhbacca So I did just what you said finally and results were a bit shocking but seem to support my concern. Top image is before setting to Do not transcode, bottom is after processing
No description
RoachForLife
RoachForLifeOP2mo ago
Why would anything still remain in the encoded-video folder at all? There are still 113gig of videos in there
schuhbacca
schuhbacca2mo ago
Not sure, what I said should remove all encoded videos that immich is tracking. Can you see what's in that folder?
RoachForLife
RoachForLifeOP2mo ago
It ended up being videos of course. Should I try to cross reference the id back to immich, or maybe to the DB? I was going to reprocess everything soon so if this is something I should do as a test I probably need to do it soon
schuhbacca
schuhbacca2mo ago
Yeah I would. Or look for errors in server log when running the transcode job with none set
RoachForLife
RoachForLifeOP2mo ago
Sorry but where do I get to the server log? Normally I just check dockge but not sure it will have the history from yesterday but ill check
schuhbacca
schuhbacca2mo ago
terminal can just run docker logs immich_server or whatever your server container name is
RoachForLife
RoachForLifeOP2mo ago
ok cool let me do that. looks like i cant go fair enough back for it all (some is cut off the screen). Is there a way to have it save this to a file maybe?
RoachForLife
RoachForLifeOP2mo ago
Here is a portion of it at least
schuhbacca
schuhbacca2mo ago
That all looks fine according to the logs (At least related to video transcoding deletion). You can try and query the db to see if any of the transcode video columns are not nulled out Have you ever re-installed or anything like that?
RoachForLife
RoachForLifeOP2mo ago
@schuhbacca Reinstalled, no, just standard updates thru the months. I am unsure how to query the db. Is there a set of instructions somewhere on this?
RoachForLife
RoachForLifeOP2mo ago
ok thanks. Regarding the query should I be looking up specific files or can I run a query to look up the null in the transcoded video column specifically? Or maybe do this? SELECT * FROM "assets" WHERE "assets"."type" = 'VIDEO';
RoachForLife
RoachForLifeOP2mo ago
No description
RoachForLife
RoachForLifeOP2mo ago
I used docker exec -it immich_postgres psql --dbname=immich --username=postgres, to get into it, which I believe is right. Doesnt seem to find the assets table It seems the table is asset not plural. Anyhow I did run this. Cant seem to copy all of it, is it possible to export to a file?
RoachForLife
RoachForLifeOP2mo ago
Anyhow heres a small snippet
No description
Sergey Katsubo
Sergey Katsubo2mo ago
the table is asset not plural.
You correctly spotted it. Plural to singular was changed in 1.136 and not yet reflected in the docs.
Sergey Katsubo
Sergey Katsubo2mo ago
Thanks for the asset snippet! 1. encodedVideoPath is null, this is good / expected as we don't transcode anymore. 2. But there are assets with originalPath containing upload/encoded-video/<...>.mp4 which is rather unexpected. Encoded-video should not appear in originalPath. It looks like transcoded video files were actually added into Immich as original videos, hmm
No description
RoachForLife
RoachForLifeOP2mo ago
@Sergey Katsubo Thanks. I do have the setting to NOT encode if a mp4 already so unsure how it would get in there. If I use this setting, should I expect zero .mp4s in the encodedvideo folder?
Sergey Katsubo
Sergey Katsubo2mo ago
Let's split concerns into 2 topics. 1. Encoded or not encoded videos / mp4. 2. The fact that ".../encoded-video/...mp4" appear as original assets. (This is troubling me) For (1) could you grab stats from the database: count of videos - by extension - and whether they have been encoded or not (if they have non-empty encodedVideoPath)
SELECT
SUBSTRING("originalFileName" FROM '\.([^\.]+)$') AS ext,
CASE
WHEN "encodedVideoPath" IS NULL
THEN NULL
ELSE 'encoded'
END AS encoded,
COUNT(*) AS count
FROM asset
WHERE type = 'VIDEO'
GROUP BY 1,2 ORDER BY 1,2;
SELECT
SUBSTRING("originalFileName" FROM '\.([^\.]+)$') AS ext,
CASE
WHEN "encodedVideoPath" IS NULL
THEN NULL
ELSE 'encoded'
END AS encoded,
COUNT(*) AS count
FROM asset
WHERE type = 'VIDEO'
GROUP BY 1,2 ORDER BY 1,2;
RoachForLife
RoachForLifeOP2mo ago
Thanks Sergey. I had moved on and deleted everything and reprocessed (see this post as it hasnt gotten any traction but is also concerning and related - https://www.reddit.com/r/immich/comments/1m8hfib/my_journey_into_immich_not_recovering_itself/ ) but I did back it up in proxmox. I will make a point of creating a new lxc restore of everything and run the sql on it and post back here. Appreciate it!
Reddit
From the immich community on Reddit
Explore this post and more from the immich community
RoachForLife
RoachForLifeOP2mo ago
Regarding backups I have the following backups, let me know which is best. I assume the one after I turned off transcoding and it removed most items but still a bunch remained in the encodedvideo folder? - Prior to any edits - After setting transcoding to OFF and it deleted nearly half of the files in encodedvideos folder
jrasm91
jrasm912mo ago
The files with -MP suffix are extracted motion videos from Android motion photos and should not be deleted.
RoachForLife
RoachForLifeOP2mo ago
@Sergey Katsubo Ok took a bit but I was able to load a new proxmox ct with the state right after I turned transcoding to OFF and it deleted a bunch, but not all, of the videos. And ran your statement. See attached
No description
RoachForLife
RoachForLifeOP2mo ago
Thanks, this is helpful. On my other machine (where I reprocessed everything, not the one I just told sergey about), it seems I have 13k motion photos videos with the -mp (40gig). I had put in a FR a month ago for the option to have Immich not process these. I obviously have these 13k where I actually would prefer the space back, and the only way I know to fix them is on my phone, meaning Id have to transfer them to and from my phone to strip the video portion. Having immich just not process the video would be nearly as good as I at least wouldnt have the extra video. It's actually kinda crazy that my entire immich library folder is 183gb and 40gb of that are motion videos I dont want 🙁
jrasm91
jrasm912mo ago
We can probably add a system config option for that.
RoachForLife
RoachForLifeOP2mo ago
Thank you for considering that! I didnt realize it was nearly 1/4 of all of my server space just in motion videos so that would be great @Sergey Katsubo Hola. Just curious if that output helped? Thanks
Sergey Katsubo
Sergey Katsubo2mo ago
Hey. Yes, it has become clearer now, thanks to the SQL statement output + your comments + Jason's note about MP being extracted motion videos. So it turned out that both my prev questions are connected, not independent:
1. Encoded or not encoded videos / mp4. 2. The fact that ".../encoded-video/...mp4" appear as original assets. (This is troubling me)
The short answer: this is the way how Immich handles MP (motion photos). It is expected. If I got it right: - Why do mp4/mov files appear in encoded-video despite Video Transcoding is disabled? (Intuitive expectation: encoded-video should be empty). Answer: because Immich extracted them from MPs and put there. This processing of MPs happens independently of Transcoding policy. - Why do mp4/mov files in encoded-video have "original" asset mark? (Intuitive expectation: encoded-videos are not originals. They are transcoded versions of actual original assets living in other locations, internal or external libraries). Answer: they appear as "original" assets because they were extracted from MPs and now live a separate live as "originals". This motion videos from MPs live in encoded-video because this is how it's implemented. @jrasm91 Is this a correct explanation? We can add an extra dimension - MP or not - to the stats from asset table in DB:
SELECT
LOWER(SUBSTRING("originalPath" FROM '\.([^\.]+)$')) AS ext,
CASE
WHEN "encodedVideoPath" IS NULL
THEN NULL
ELSE 'encoded'
END AS encoded,
CASE
WHEN "originalPath" LIKE '%-MP.%'
THEN 'MP'
ELSE NULL
END AS mp,
COUNT(*) AS count
FROM asset
WHERE type = 'VIDEO'
GROUP BY 1, 2, 3 ORDER BY 1, 2, 3;
SELECT
LOWER(SUBSTRING("originalPath" FROM '\.([^\.]+)$')) AS ext,
CASE
WHEN "encodedVideoPath" IS NULL
THEN NULL
ELSE 'encoded'
END AS encoded,
CASE
WHEN "originalPath" LIKE '%-MP.%'
THEN 'MP'
ELSE NULL
END AS mp,
COUNT(*) AS count
FROM asset
WHERE type = 'VIDEO'
GROUP BY 1, 2, 3 ORDER BY 1, 2, 3;
RoachForLife
RoachForLifeOP2mo ago
No description
RoachForLife
RoachForLifeOP2mo ago
@Sergey Katsubo Here is the latest sql query. I am not certain this all is making sense tho. So in my live production Immich which has a lot more data on it (reprocessed everything after turning transcoding off). There I have 40gb of Motion Photo videos (with -mp). On this other box we are using, right when I disabled transcoding, it has about 113gb of space being used in the encodedvideo folder. That leaves a difference of about 73gb of space that is not a motion photo video and should not exist. I am re-running to see how many MP files exist right now in the previous state system I am working on with you and will report back in 10min when it goes through its check
RoachForLife
RoachForLifeOP2mo ago
Can you confirm that if transcoding is OFF, that, less the MP files, it should clear out all of the videos? The system does have this setting on currently
No description
RoachForLife
RoachForLifeOP2mo ago
So if something is a h.264, does it just use the file in the external library or does it make its own copy for the encodedvideo folder? I suppose if the answer to that is, it makes its own copy (which feels redundant to not just reference the same file over on my external library) but anyhow, then would unchecking H.264 and setting to DONT TRANSCODE, would it then have nothing (zero) in the encoded video folder? Ultimately, reprocessing the exact same data has produced an encodedvideo folder that is 30-40gb smaller so something extra was definitely lingering in there
jrasm91
jrasm912mo ago
Yes, we need the motion video portion of the live photo to be it's own asset with an original path, but that original asset is actually generated from the android motion jpeg. So it's an original path, but generated. So we decided to put it in encoded-videos, to indicate that it is generated.
Sergey Katsubo
Sergey Katsubo2mo ago
https://github.com/immich-app/immich/blob/ce2ea989265e878eedf4aaf1e1b73802b1c53c21/server/src/services/media.service.ts#L492-L499 - It removes the transcode when TranscodePolicy.Disabled - It writes a log message at default LOG level, so it will be visible in the Immich container logs. Did you run 1 pass of reprocessing (Transcode Videos job) or 2 passes and the second pass freed additional space?
Immich
Immich2mo ago
if (target === TranscodeTarget.None && !this.isRemuxRequired(ffmpeg, format)) {
if (asset.encodedVideoPath) {
this.logger.log(\`Transcoded video exists for asset ${asset.id}, but is no longer required. Deleting...\`);
await this.jobRepository.queue({ name: JobName.FileDelete, data: { files: [asset.encodedVideoPath] } });
await this.assetRepository.update({ id: asset.id, encodedVideoPath: null });
} else {
this.logger.verbose(\`Asset ${asset.id} does not require transcoding based on current policy, skipping\`);
}
if (target === TranscodeTarget.None && !this.isRemuxRequired(ffmpeg, format)) {
if (asset.encodedVideoPath) {
this.logger.log(\`Transcoded video exists for asset ${asset.id}, but is no longer required. Deleting...\`);
await this.jobRepository.queue({ name: JobName.FileDelete, data: { files: [asset.encodedVideoPath] } });
await this.assetRepository.update({ id: asset.id, encodedVideoPath: null });
} else {
this.logger.verbose(\`Asset ${asset.id} does not require transcoding based on current policy, skipping\`);
}
Sergey Katsubo
Sergey Katsubo2mo ago
From the last query we see that there do exist a lot of videos having their transcoded counterpart (marked as "encoded" on the screenshot). Supposedly they should be removed (eventually?) when transcode policy is OFF. Unless they do not match criteria for removal, e.g. isRemuxRequired from the linked source code. This is how I understand it, though reality can be surprising.
RoachForLife
RoachForLifeOP2mo ago
@Sergey Katsubo tbh, I dont recall how many times I processed. I suppose I could hit process again now and it would guarantee it ran at least 2x? Im guessing I ran it just once after switching to dont transcode and then it took it bit but began dropping in space
RoachForLife
RoachForLifeOP2mo ago
Also if it matters, I guess this version (before reprocessing) had a lot more MP than after reprocessing, which is also odd since the data it pulls from (my ext library) is identical. Anyhow here is what it has
No description
RoachForLife
RoachForLifeOP2mo ago
Whereas its more like 40gb in its reprocessed state (basically started over after deleting the encodedvideo folder and turning transcoding back on) Im wondering now if part of the issue of Immich not recovering its data after deleting items is more centered around these MP video files. The difference in MPs is 65gb which is very close to the total difference between the old system (before even turning off transcoding) and after I reprocessed everything. I wonder if Immich does not properly delete MP videos when an item is deleted from the external library?
Sergey Katsubo
Sergey Katsubo2mo ago
if Immich does not properly delete MP videos when an item is deleted from the external library?
Yeah, that's possible. For example these two open bugs are about external library and live photos, albeit they are different from your issue: - https://github.com/immich-app/immich/issues/17731 - https://github.com/immich-app/immich/issues/20031
Immich
Immich2mo ago
[Issue] MOV files left behind if deleted while External LIbrary Watching enabled (immich-app/immich#17731) [Issue] Live Photo Video and Sidecar getting deleted when renaming external library folders (immich-app/immich#20031)
RoachForLife
RoachForLifeOP2mo ago
@Sergey Katsubo I suppose I would ask if more investigation could be done. Not so much for my issue as clearing everything and reprocessing has fixed the issue (for now) but to make the product better long term. I would not want to HAVE to delete everything and reprocess all just to fix these types of issues every so often. Having immich correctly deal with the deleted files in its DB would be a far superior solution. I am sure there are lots out there that have these orphaned files sitting in their library folder and don't even realize it. I only took a vested interest as my 500gb SSD was climbing fast and not ever going down which prompted me to look into it more. If there is anything more I can do on my end to assist I am happy to, especially if it leads to some improvements to the product. One thought would be a utility of sorts that could examine the DB vs library files vs external library, so compare and possibly clean itself up? Kinda like a reprocess but without having to truely delete things from scratch but more just to clean/delete only the stuff it needs to. I say this as a non dev so no idea how difficult this is
jrasm91
jrasm912mo ago
Internally we have already talked at length about building more "integrity checks" into the application. The idea is to basically convert external library scanning into general file scanning on a per library basis to verify that file paths in the database correspond to a corresponding files on disk and that checksums match between the two. This would lead to detecting differences. Either files without database records or database records with corresponding files. Or records that mismatch. But this is not an easy or trivial change and will require significant engineering effort to implement, but it is pretty high on our list.
RoachForLife
RoachForLifeOP4w ago
@jrasm91 Thank you so much for the candid information. I'm happy to hear this. Yeh, I am definitely not a dev (although I work with developers in my day job) so I get it. Been using Immich nearly a year now, and its been wonderful, and hope to continue using it for many more. I do feel like a lot of the users (don't have any metrics to support this just based on reddit post volume) use external libraries with immich as the front end, so things like this go a long way to supporting that. Anyhow thanks again, and I will save these Proxmox CTs so if I can be of help checking stuff I am happy to help. PS - Thanks also on your mention the other day on considering an option to disable Motion Videos all together. If my issue is linked to MP it may resolve that just by disabling it (assuming it cleans up the left behind ones of course). Anyhow, all good stuff, thanks guys! @js' @jrasm91 If it makes it any easier, I had put in a FR for the motion photo thing, url is https://github.com/immich-app/immich/discussions/19560 . I've began taking notice more of the devs on each release and see you have added so much towards Immich so just wanted to say thank you again for the amazing product you and the team have built
Immich
Immich4w ago
[Discussion] [Feature] Ability to ignore motion photos when transcoding (immich-app/immich#19560)
RoachForLife
RoachForLifeOP4w ago
@jrasm91 I guess if you have a minute at some point and can take a look at this post on reddit I did. I am also exploring manually updating the photo itself to remove the motion. Its not my preference since I am okay with the file size being larger on my external library (NAS) for the original file but trying to reduce what is stored on the immich library side (on a SSD). Anyhow I was trying to confirm what should happen if I were to update these, perhaps you would know? https://www.reddit.com/r/immich/comments/1mqumai/what_will_happen_if_i_strip_an_already_processed/ Still definitely my preference for an option in the settings to just disable processing of Motion Photos all together but exploring all options. Thanks again!
Reddit
From the immich community on Reddit
Explore this post and more from the immich community

Did you find this page helpful?