Pods suddenly start with cuda 12.4 when 12.8 is requested

I have been starting pods via the python sdk and specified the cuda version that I need (12.8). This used to work well until a day or two ago. Today, when I start pods, I get this in the logs:

==========

== CUDA ==

==========

CUDA Version 12.4.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.

By pulling and using the container, you accept the terms and conditions of this license:

https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

==========

== CUDA ==

==========

CUDA Version 12.4.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.

By pulling and using the container, you accept the terms and conditions of this license:

https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

I also can't ssh into them and the entrypoint script does not execute healthily. Example pod id: fnb9ae7p2wvgwo

Runpod•5mo ago•

1 reply

nielsrolf

Pods suddenly start with cuda 12.4 when 12.8 is requested

I have been starting pods via the python sdk and specified the cuda version that I need (12.8). This used to work well until a day or two ago. Today, when I start pods, I get this in the logs:

==========

== CUDA ==

==========

CUDA Version 12.4.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.

By pulling and using the container, you accept the terms and conditions of this license:

https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

==========

== CUDA ==

==========

CUDA Version 12.4.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.

By pulling and using the container, you accept the terms and conditions of this license:

https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

I also can't ssh into them and the entrypoint script does not execute healthily. Example pod id: fnb9ae7p2wvgwo

Pods suddenly start with cuda 12.4 when 12.8 is requested

Similar Threads

Pods suddenly start with cuda 12.4 when 12.8 is requested

Similar Threads

Similar Threads

Similar Threads