Runpod•4mo ago

When encoding video with ffmpeg, nvenc does not work.

DC:US-NC-1
GPU:RTX 5090

I have switched data centers to US-IL-1 in addition to US-NC-1, but the results remain the same.

cmd

-f lavfi -i testsrc=duration=5:size=1280x720:rate=30 -c:v h264_nvenc -pix_fmt yuv420p -t 5 /tmp/test.mp4 -y

-f lavfi -i testsrc=duration=5:size=1280x720:rate=30 -c:v h264_nvenc -pix_fmt yuv420p -t 5 /tmp/test.mp4 -y

ffmpeg version 7.1.1 Copyright (c) 2000-2025 the FFmpeg developers
  built with gcc 13 (Ubuntu 13.3.0-6ubuntu2~24.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-alsa --enable-cuda-llvm --enable-cuvid --enable-ffprobe --enable-gpl --enable-libaom --enable-libass --enable-libdav1d --enable-libfdk_aac --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-libkvazaar --enable-liblc3 --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libplacebo --enable-librav1e --enable-librist --enable-libshaderc --enable-libsrt --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libvvenc --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-nonfree --enable-nvdec --enable-nvenc --enable-opencl --enable-openssl --enable-stripping --enable-vaapi --enable-vdpau --enable-version3 --enable-vulkan
  libavutil      59. 39.100 / 59. 39.100
  libavcodec     61. 19.101 / 61. 19.101
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample   5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
Input #0, lavfi, from 'testsrc=duration=5:size=1280x720:rate=30':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, rgb24, 1280x720 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 30 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> h264 (h264_nvenc))
Press [q] to stop, [?] for help
[h264_nvenc @ 0x5b844ea0d440] OpenEncodeSessionEx failed: unsupported device (2): (no details)
[h264_nvenc @ 0x5b844ea0d440] No capable devices found
[vost#0:0/h264_nvenc @ 0x5b844ea0fdc0] Error while opening encoder - maybe incorrect parameters such as bit_rate, rate, width or height.
[vf#0:0 @ 0x5b844ea2afc0] Error sending frames to consumers: Generic error in an external library
[vf#0:0 @ 0x5b844ea2afc0] Task finished with error code: -542398533 (Generic error in an external library)
[vf#0:0 @ 0x5b844ea2afc0] Terminating thread with return code -542398533 (Generic error in an external library)
[vost#0:0/h264_nvenc @ 0x5b844ea0fdc0] Could not open encoder before EOF
[vost#0:0/h264_nvenc @ 0x5b844ea0fdc0] Task finished with error code: -22 (Invalid argument)
[vost#0:0/h264_nvenc @ 0x5b844ea0fdc0] Terminating thread with return code -22 (Invalid argument)
[out#0/mp4 @ 0x5b844ea0f540] Nothing was written into output file, because at least one of its streams received no packets.
frame=    0 fps=0.0 q=0.0 Lsize=       0KiB time=N/A bitrate=N/A speed=N/A
Conversion failed!

ffmpeg version 7.1.1 Copyright (c) 2000-2025 the FFmpeg developers
  built with gcc 13 (Ubuntu 13.3.0-6ubuntu2~24.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-alsa --enable-cuda-llvm --enable-cuvid --enable-ffprobe --enable-gpl --enable-libaom --enable-libass --enable-libdav1d --enable-libfdk_aac --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-libkvazaar --enable-liblc3 --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libplacebo --enable-librav1e --enable-librist --enable-libshaderc --enable-libsrt --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libvvenc --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-nonfree --enable-nvdec --enable-nvenc --enable-opencl --enable-openssl --enable-stripping --enable-vaapi --enable-vdpau --enable-version3 --enable-vulkan
  libavutil      59. 39.100 / 59. 39.100
  libavcodec     61. 19.101 / 61. 19.101
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample   5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
Input #0, lavfi, from 'testsrc=duration=5:size=1280x720:rate=30':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, rgb24, 1280x720 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 30 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> h264 (h264_nvenc))
Press [q] to stop, [?] for help
[h264_nvenc @ 0x5b844ea0d440] OpenEncodeSessionEx failed: unsupported device (2): (no details)
[h264_nvenc @ 0x5b844ea0d440] No capable devices found
[vost#0:0/h264_nvenc @ 0x5b844ea0fdc0] Error while opening encoder - maybe incorrect parameters such as bit_rate, rate, width or height.
[vf#0:0 @ 0x5b844ea2afc0] Error sending frames to consumers: Generic error in an external library
[vf#0:0 @ 0x5b844ea2afc0] Task finished with error code: -542398533 (Generic error in an external library)
[vf#0:0 @ 0x5b844ea2afc0] Terminating thread with return code -542398533 (Generic error in an external library)
[vost#0:0/h264_nvenc @ 0x5b844ea0fdc0] Could not open encoder before EOF
[vost#0:0/h264_nvenc @ 0x5b844ea0fdc0] Task finished with error code: -22 (Invalid argument)
[vost#0:0/h264_nvenc @ 0x5b844ea0fdc0] Terminating thread with return code -22 (Invalid argument)
[out#0/mp4 @ 0x5b844ea0f540] Nothing was written into output file, because at least one of its streams received no packets.
frame=    0 fps=0.0 q=0.0 Lsize=       0KiB time=N/A bitrate=N/A speed=N/A
Conversion failed!

This is the result I got running on my RTX 4090. There are no issues with the container image and command.

docker run --rm -it --gpus=all \
                     -v $(pwd):/config \
                     linuxserver/ffmpeg:7.1.1 \
                     -hwaccel cuda -hwaccel_device 0 -f lavfi -i testsrc=duration=5:size=1280x720:rate=30 -c:v h264_nvenc -pix_fmt yuv420p -t 5 /tmp/test.mp4 -y
  ...
        encoder         : Lavc61.19.101 h264_nvenc
      Side data:
        cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: N/A
[out#0/mp4 @ 0x619f0fd1afc0] video:196KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 1.333094%
frame=  150 fps=0.0 q=8.0 Lsize=     199KiB time=00:00:04.90 bitrate= 332.2kbits/s speed=27.8x

docker run --rm -it --gpus=all \
                     -v $(pwd):/config \
                     linuxserver/ffmpeg:7.1.1 \
                     -hwaccel cuda -hwaccel_device 0 -f lavfi -i testsrc=duration=5:size=1280x720:rate=30 -c:v h264_nvenc -pix_fmt yuv420p -t 5 /tmp/test.mp4 -y
  ...
        encoder         : Lavc61.19.101 h264_nvenc
      Side data:
        cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: N/A
[out#0/mp4 @ 0x619f0fd1afc0] video:196KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 1.333094%
frame=  150 fps=0.0 q=8.0 Lsize=     199KiB time=00:00:04.90 bitrate= 332.2kbits/s speed=27.8x

Dj•9/9/25, 5:43 AM

NVENC and ffmpeg are very very sensitive to your nodes specific driver version. When you're deploying your pod, used the advanced filter to select CUDA 12.8 and 12.9.

DDj NVENC and ffmpeg are very very sensitive to your nodes specific driver version. ...

Azunyan1111OP•9/9/25, 10:20 AM

I have already made that specification.
It was clear that it would not work with either 12.8 or 12.9.
I built ffmpeg from scratch inside the container, but that made no difference.
None of the ffmpeg versions I tried worked.

Please tell me which data centers have an RTX 5090 where NVENC works.
If it is my issue, I will gladly take care of it.

Dj•9/9/25, 4:03 PM

NVENC is a chip on the graphics card included on every NVIDIA device produced after 2006. Have you tried any other ffmpeg version (8.0+) or any of the mainline alternative builds?

https://github.com/BtbN/FFmpeg-Builds/releases/tag/latest

Youll want ffmpeg-master-latest-linux64-lgpl.tar.xz. You can extract this with

GitHub

Release Latest Auto-Build (2025-09-09 13:41) · BtbN/FFmpeg-Builds

DDj NVENC is a chip on the graphics card included on every NVIDIA device produced af...

riverfog7•9/9/25, 4:32 PM

from what I know the AI training chips (A100 H100) doesn't have NVENC

riverfog7•9/9/25, 4:32 PM

only NVDEC

riverfog7•9/9/25, 4:33 PM

but all consumer & worktation grade cards do

riverfog7•9/9/25, 4:33 PM

https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new

NVIDIA Developer

Video Encode and Decode GPU Support Matrix

Find the related video encoding and decoding support for all NVIDIA GPU products.

riverfog7•9/9/25, 4:40 PM

works with RTX 2000 Ada

riverfog7•9/9/25, 4:42 PM

doesn't work with 5090

riverfog7•9/9/25, 4:42 PM

riverfog7•9/9/25, 4:47 PM

same error with community cloud

riverfog7•9/9/25, 4:48 PM

https://obsproject.com/forum/threads/obs-not-working-with-rtx-5090-nv_enc_err_invalid_device.184606/

OBS Forums

OBS not working with RTX 5090 (NV_ENC_ERR_INVALID_DEVICE)

Hey!
I'm not entirely sure if this is going to be an isolated case or a bigger issue across other RTX 5090 Cards.
I'll gladly help find a solution for this & hopefully will help find a fix for other 50 Cards users as well.

I have spent over a week working around with OBS in an attempt to fix...

riverfog7•9/9/25, 4:48 PM

looks like a driver issue

DDj NVENC is a chip on the graphics card included on every NVIDIA device produced af...

Azunyan1111OP•9/9/25, 9:28 PM

Yes. I have tried all the items you pointed out and have confirmed that it is impossible to execute them.
Of course, I am also using ffmpeg-master-latest-linux64-lgpl.tar.xz.

Rriverfog7 looks like a driver issue

Azunyan1111OP•9/9/25, 9:32 PM

No, I don’t think so. This is a data center issue.
On the RTX 4090, the same error occurs in US-IL-1, EUR-NO-1, and EUR-IS-2, but encoding works normally on EU-RO-1.

Also, some RTX 5090s have been confirmed to work. However, it’s no longer possible to get assigned to that pod.

riverfog7•9/10/25, 1:44 AM

That's strange

riverfog7•9/10/25, 1:45 AM

But i heard that runpod is using early driver versions

riverfog7•9/10/25, 1:45 AM

So I thought it was a driver issue

Rriverfog7 That's strange

Azunyan1111OP•9/10/25, 4:12 AM

I will present evidence that supports my claim.
This is a serverless endpoint that runs the command

using the image on a serverless worker.

The bad workers show the same error I reported, while the properly functioning workers correctly display encoding speed logs.
Therefore, this cannot be concluded as a driver issue, since there are GPU servers that operate normally.

The four attached screenshots are logs from the workers that function correctly.
The fifth image shows a list of both healthy and unhealthy workers. (All unhealthy ones output the error I posted and then stopped processing.)

And as an important detail, I have confirmed that there is no difference in the Driver Version between the bad workers and the healthy ones.
Therefore, this is a data center issue.

DDj NVENC and ffmpeg are very very sensitive to your nodes specific driver version. ...

Azunyan1111OP•9/10/25, 4:16 AM

I have presented evidence that this is a data center issue. Please investigate. I would appreciate it if you could escalate the ticket.

riverfog7•9/10/25, 4:31 AM

this is strange

riverfog7•9/10/25, 4:34 AM

NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 12.9: work (EU-RO-1)
NVIDIA-SMI 570.172.08 Driver Version: 570.172.08 CUDA Version: 12.8 : no work (EUR-IS-2)
NVIDIA-SMI 570.144 Driver Version: 570.144 CUDA Version: 12.8: no work (EUR-IS-1)
NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 12.9 : no work (EU-RO-1)
NVIDIA-SMI 570.144 Driver Version: 570.144 CUDA Version: 12.8: work (EUR-IS-1)
NVIDIA-SMI 570.144 Driver Version: 570.144 CUDA Version: 12.8: work (EUR-IS-1)
NVIDIA-SMI 570.144 Driver Version: 570.144 CUDA Version: 12.8 : no work (EUR-IS-1)

riverfog7•9/10/25, 4:34 AM

it doesn't depend on driver version?

riverfog7•9/10/25, 4:34 AM

seems like EUR-RO-1 uses newer drivers

riverfog7•9/10/25, 4:35 AM

but it sometimes doesn't work there too

Rriverfog7 it doesn't depend on driver version?

Azunyan1111OP•9/10/25, 4:51 AM

Yes. NVENC reacts sensitively to things like driver version, but even with exactly the same version, in exactly the same DC, and with exactly the same configuration, differences in behavior can be observed.
As you know, EUR-IS-2 is 570.172.08, but there are workers operating normally on 570.172.08, and there are also bad workers on 570.172.08.
The attached image shows information from a worker that operated normally.

riverfog7•9/10/25, 4:52 AM

Yeah I can confirm

riverfog7•9/10/25, 4:52 AM

And i didnt know that runpod dashboard shows driver version

AAzunyan1111 I have presented evidence that this is a data center issue. Please investigate. ...

Dj•9/10/25, 9:37 AM

I really appreciate you looking into this. I'll have this escalated.

Rriverfog7 NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 1...

Dj•9/10/25, 9:37 AM

Are you able to share the ids of these deployments or no? It's okay if not I can get it all manually.

riverfog7•9/10/25, 9:47 AM

8sv5k7ublivjhq

riverfog7•9/10/25, 9:47 AM

should be the serverless endpoint ID

DDj I really appreciate you looking into this. I'll have this escalated.

Azunyan1111OP•9/12/25, 5:01 AM

Thank you very much. Since I would like to track the situation, please escalate from this thread and issue a ticket.

Azunyan1111OP•9/12/25, 5:03 AM

skwe5i0dbvkzeu
This is the serverless endpoint ID that I presented as evidence.

AAzunyan1111 skwe5i0dbvkzeu This is the serverless endpoint ID that I presented as evidence.

PoddyAPP•9/12/25, 11:25 AM

@KaSuTeRaMAX

Escalated To Zendesk

The thread has been escalated to Zendesk!

Ticket ID: #23406

neutel_•9/14/25, 10:02 PM

Any news on this issue?

Nneutel_Any news on this issue?

Azunyan1111OP•9/15/25, 3:31 AM

In the support ticket, they reported:

It seems they are currently investigating.

I will share any new information here as soon as it becomes available.

Azunyan1111OP•9/17/25, 3:14 AM

Runpod sincerely conducted additional investigation and provided support.
I will share the details of the ticket.

----------------------------------------------------------------------------------------------------------

Following up on your request, we’ve reproduced the NVENC failure you reported across multiple regions and GPU types. After investigation, we’ve classified this as an upstream issue with FFmpeg/NVIDIA. Specifically, the problem appears to stem from how device indices are mapped inside containers (when /dev/nvidia* devices don’t align with nvidia-smi indices).
This behavior matches several active upstream reports:
https://trac.ffmpeg.org/ticket/11694
https://github.com/NVIDIA/nvidia-container-toolkit/issues/1249
https://github.com/NVIDIA/nvidia-container-toolkit/issues/1209
https://github.com/NVIDIA/nvidia-container-toolkit/issues/1197
https://github.com/NVIDIA/k8s-device-plugin/issues/1282
Since the root cause lies upstream, a permanent fix will need to come from the FFmpeg/NVIDIA teams. That said, we’ll continue to monitor developments closely and will keep you updated on any relevant progress or workarounds.

----------------------------------------------------------------------------------------------------------

flexgrip•10/11/25, 6:49 AM

I am seeing the same issue @KaSuTeRaMAX. Thank god I found this. I thought I was going crazy.

Seems to be a random roll of the dice on whether a pod will work or not. That being said, I have not seen any of these errors on the serverless endpoints.

I wonder if it is safe to depend on the serverless endpoints for large batches of requests or if this only affects the pods?

flexgrip•10/11/25, 6:50 AM

I will read the issues mentioned in the runpod support response. Perhaps we can just ship our own binaries that have a fix.

flexgrip•10/11/25, 7:10 AM

Seeing a lot of talk about the 570 driver being the culprit. When I launch RTX PRO 6000 pods I always get nvidia driver version >= 580 and have yet to see the issue on there. But that could be anecdotal.

It seems there is nothing we can do to fix the problem itself. But I am going to try getting the and passing that into ffmpeg with

Maybe that will work. Going to bed but will continue tinkering with it tomorrow.

Rriverfog7 NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 1...

riverfog7•10/11/25, 7:30 AM

this might support that clain

riverfog7•10/11/25, 7:31 AM

only servers with driver version 570 didn't work

flexgrip•10/11/25, 5:45 PM

Yeah as I understand the issue, it's with the way multiple gpu servers start the docker container. Whatever fancy way runpod spins them up, they are probably doing something like if the first gpu is already being used in another container.

As I found out last night, you can spin up the exact same image over and over on the same gpu type. Seems like unless you get gpu 0, you get this problem. The thing is, I have not seen this issue even once with RTX PRO 6000 machines.

flexgrip•10/11/25, 5:45 PM

I'm gonna see right now if I can reproduce the issue, and then work around it by specifying which device to enumerate in ffmpeg

flexgrip•10/11/25, 6:16 PM

seems to have some effect. At least I get a different error when I set different device ids.

Just for fun, I tried symlinking /dev/nvidia1 to /dev/nvidia0 and it had no effect.

I am kind of out of ideas on how to fix this. It feels like if you get a set of circumstances, you just can't work around it.

You select a gpu pod that isn't having all the gpus passed to it
Someone else is using gpu 0
You are on driver ~570 (although I havent confirmed this)

flexgrip•10/11/25, 6:17 PM

Curious why I never saw this on the serverless endpoints. That's what I indend to use anyway so if it works there, then no big deal. But I can't really finish my project if there's a random chance each serverless endpoint worker might fail too.

flexgrip•10/11/25, 6:21 PM

Btw. I just launched a pod and got the same exact scenario where I got /dev/nvidia1 but since it is driver 580, it seems to work just fine

riverfog7•10/11/25, 6:37 PM

So the only workaround is using a dofferent gpu?