Runpod•9mo ago

Struggling with runpod unable to access htpp server or terminal error.

Hey guys! I have posted this same issue some time ago, but I cannot use any of the pods at all when I try to run Kobold AI with Fallen Llama or any other Model.

The first time I used this I been able to access runpod normally, but after a few days I am dealing with the runpod issue ever since.

I have this for my screenshots.

Arsen32OP•4/6/25, 6:50 PM

Currently switching to a 2x RTX 4000 Ada to see if a lack of GPU's might be the issue.

Edit: Wait nvm still same crap.

Arsen32OP•4/6/25, 6:53 PM

Jason•4/7/25, 12:23 AM

It says oom

Jason•4/7/25, 12:24 AM

Oom means you need better gpus with more ram

Jason•4/7/25, 12:24 AM

Right @Henky!!

Henky!!•4/7/25, 12:24 AM

Out of memory indeed

Henky!!•4/7/25, 12:25 AM

Without the rest of the log its hard to say

Jason•4/7/25, 12:25 AM

Which is fallen llama tho

Jason•4/7/25, 12:25 AM

Oh it's a custom llama 3

Henky!!•4/7/25, 12:25 AM

Its a 70B

Jason•4/7/25, 12:25 AM

Henky!!•4/7/25, 12:25 AM

Which will fit on a single A40/A6000 if its Q4_K_S / Q4_K_M

Henky!!•4/7/25, 12:26 AM

I suspect a to large quant was used

Henky!!•4/7/25, 12:30 AM

Verified the model can run

Henky!!•4/7/25, 12:31 AM

Its apparently a Deepseek 70B Distill finetune

Henky!!•4/7/25, 12:31 AM

Kobold auto picked it up as deepseek

Henky!!•4/7/25, 12:31 AM

Link I used was https://huggingface.co/TheDrummer/Fallen-Llama-3.3-R1-70B-v1-GGUF/resolve/main/L33-Tiger-R1-70B-v1b-Q4_K_M.gguf?download=true , on a single A40

Henky!!•4/7/25, 12:32 AM

Its quite close so if you want more than 4K context you will have to go higher than the 1xA40

HHenky!!Which will fit on a single A40/A6000 if its Q4_K_S / Q4_K_M

riverfog7•4/7/25, 1:56 AM

With short context lengths yes

HHenky!!Its quite close so if you want more than 4K context you will have to go higher t...

Arsen32OP•4/7/25, 3:55 AM

So my problem could be solved by potentially using max GPUs available and after that it would get the http running?

AArsen32 So my problem could be solved by potentially using max GPUs available and after ...

Henky!!•4/7/25, 3:56 AM

You need 48GB for Q4 at 4K on 70B's, you only went up to 32GB so it crashed

AArsen32 So my problem could be solved by potentially using max GPUs available and after ...

riverfog7•4/7/25, 3:56 AM

try 2xA40 first

Rriverfog7 try 2xA40 first

Arsen32OP•4/8/25, 5:47 AM

Hey river! I have finally had some time to get this done, and I switched to 2xA40. It did load everything, and the terminal started working, but the http service still didn't work. I ended up deleting the pod because I don't want to spend too much to maintain this.

Here's the text logs.

logs.txt475.44KB

AArsen32 Hey river! I have finally had some time to get this done, and I switched to 2xA4...

riverfog7•4/8/25, 6:06 AM

The log is cut tho

riverfog7•4/8/25, 6:06 AM

only shows the installing part

Arsen32OP•4/8/25, 6:09 AM

Dang

Arsen32OP•4/8/25, 6:09 AM

Can't access the pod to re-download now coz I got rid of it lol

riverfog7•4/8/25, 6:10 AM

lol

Arsen32OP•4/8/25, 6:12 AM

Do you think the http service not ready issue I had earlier was mainly because of low ram or something? Happened for so many pods. I thought fixing the out of memory issue would deal with it but I got the same Http service is not ready issue

riverfog7•4/8/25, 6:14 AM

what was your model context length

riverfog7•4/8/25, 6:14 AM

like the one you set

Arsen32OP•4/8/25, 6:15 AM

I wasn't sure. Didn't pay attention tbh

Arsen32OP•4/8/25, 6:15 AM

Just chose my template and the model and thought things would work out since they did before

AArsen32 Do you think the http service not ready issue I had earlier was mainly because o...

Jason•4/8/25, 8:10 AM

Check your logs to figure out why the program isn't running, that's why port isn't ready

AArsen32 Just chose my template and the model and thought things would work out since the...

Henky!!•4/8/25, 10:47 AM

The log cuts off right before the context portion, either you didn't wait long enough or you put to much context and overflowed the GPU. How much context did you put?

Henky!!•4/8/25, 10:50 AM

Or wait I can see it

Henky!!•4/8/25, 10:50 AM

You put 65K context, which is probably to much for 2xA40

Henky!!•4/8/25, 10:51 AM

The single A40 could only handle 4K context at Q4_K_S

Henky!!•4/8/25, 10:52 AM

And for most normal use cases your never going to hit 65K

Henky!!•4/8/25, 11:01 AM

Interestingly 65536 does fit for me on the Q4_K_S link I posted

Henky!!•4/8/25, 11:08 AM

Seems to fit in Q5_K_S as well so I think you actuallly shut it down right before it would have finished loading

Henky!!•4/8/25, 11:09 AM

The main reason it took a while is that whatever pod you rented didn't have great download speeds, secure cloud downloads a lot faster

Henky!!•4/8/25, 11:10 AM

The http only comes online once its 100% loaded

HHenky!!Or wait I can see it

riverfog7•4/8/25, 11:52 AM

how did u see it tho

Henky!!•4/8/25, 11:52 AM

Theres a line with all the parameters it got assigned

riverfog7•4/8/25, 11:53 AM

oh you hav to download it

riverfog7•4/8/25, 11:53 AM

lol

riverfog7•4/8/25, 11:53 AM

im stupid

Henky!!•4/8/25, 11:53 AM

It was likely 1 minute away from working

HHenky!!It was likely 1 minute away from working

riverfog7•4/8/25, 11:54 AM

this

Struggling with runpod unable to access htpp server or terminal error.

Similar Threads

Similar Threads

Similar Threads