OutOfMemory
why my tasks keeps on failing with
out of memory
I'm just running large-v2 on faster-whisper on a 4090 GPU19 Replies
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
it was running so well the past few weeks this started happening
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I'm using my own docker image that's been running for 4 months now
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
the thing is when I start a new task the GPU mermory indicator shows a 98% usage
to fix this I have to puit the max workers to 0 and wait
then put them back up
I'm not using FlashBoot
this acts as if FlashBoot is ON
this is still going ..
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Same issue here, trying to deploy llama-3-70B and other LLM, all are erroring out with OutOfMemory error. Even when using the highest GPU tier.
yes opened a ticket
after contacting the support it seems like I was loading the model inside the handler function (it should be done outside the function)
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I updated my image and I'm gonna run it for a few days
when there is high traffic that's when it happens
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
thanks !!
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I'm running into the same issue, started getting OOM errors the past few weeks. No code change. I contacted support but got no reply yet
load your model before entering the handler function
I think now FlashBoot(?) runs by default
@Théo Champion
I do load my models outside the handler function
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View