Runpod•2y ago

Safetensor safeopen OS Error device not found

Running inference on severless endpoint and this line of code:

with safetensors.safe_open(path, framework="pt", device="cpu") as f:

with safetensors.safe_open(path, framework="pt", device="cpu") as f:

with safetensors.safe_open(path, framework="pt", device="cpu") as f:

with safetensors.safe_open(path, framework="pt", device="cpu") as f:

Throws

OSError: No such device (os error 19)

OSError: No such device (os error 19)

OSError: No such device (os error 19)

OSError: No such device (os error 19)

Running on RTX5000. Attached a network volume. The path used in the

safe_open

safe_open

safe_open

safe_open leads to a safetensor file in

/runpod-volume/example.safetensor

/runpod-volume/example.safetensor

/runpod-volume/example.safetensor

/runpod-volume/example.safetensor

Anyone got this error before?

ashley•2/16/24, 8:59 AM

Are you using network storage?

Aashley Are you using network storage?

HelloOP•2/16/24, 9:00 AM

Yes I am! I have attached it to the runpod serverless instance under the advanced settings as well.

ashley•2/16/24, 9:01 AM

Sounds like it could be an issue with unstable network storage:

I've seen that happen when using network mounted disks.

If the network is flaky then the read might fail even though the rest went fine. Error should be transient though.
Could that be it ?

I've seen that happen when using network mounted disks.

If the network is flaky then the read might fail even though the rest went fine. Error should be transient though.
Could that be it ?

Aashley Sounds like it could be an issue with unstable network storage: ``` I've seen th...

HelloOP•2/16/24, 9:06 AM

I see. To be honest, I have restarted it multiple time and even managed to print out the contents of the network mounted disk. I guess I could try it without a network disk and update this post accordingly.

ashley•2/16/24, 9:07 AM

Yeah worth a shot to try and figure out where its going wrong.

HelloOP•2/16/24, 9:21 AM

Just tried, same error haha.

ashley•2/16/24, 9:24 AM

What is the path to the file you are trying to open?

HelloOP•2/16/24, 9:32 AM

I am trying to open

/runpod-volume/example.safetensor

/runpod-volume/example.safetensor

. Note that I didn't rename runpod-volumerunpod-volume to something else since I assume runpod-volumerunpod-volume is just a folder with or without a network disk mounted.

ashley•2/16/24, 9:36 AM

Does the folder exist without a network volume though? Your network volume gets mounted there automatically by serverless if you attach a network volume to your endpoint, but it doesn't exist without a network volume unless you create it.