Runpod•2mo ago

Stuck on "Not Ready" - Skill issue?

Hi all, I'm new to the forum but love RunPod - great product. Currently trying to deploy a production service and require a bit of guidance. I have a custom worker running FastAPI on Port 80 (health port 80) - I did this by letting it default. I can see from the worker logs that the FastAPI does boot. I can even see /ping being received with a 200 response code. However I can't ping it myself, or access any custom endpoints. The services seem to be stuck on "Not Ready" and workers reboot after a few minutes (roughly 10) . I suspect skill issue! Details: - 80 port exposed on Docker configuration by default. - FastAPI confirmed it is listening on that port. - Based on this template: https://github.com/runpod-workers/worker-load-balancing/tree/main (literally the same app.py) If you have any ideas on what it could be, I'd greatly appreciate any assistance. Have a great day.

GitHub

GitHub - runpod-workers/worker-load-balancing: A Runpod worker temp...

A Runpod worker template for load balancing Serverless endpoints. - runpod-workers/worker-load-balancing

38 Replies

Unknown User•2mo ago

Message Not Public

huwprosserOP•2mo ago

My apologies. The HTTP service.

huwprosserOP•2mo ago

It never turns green. The same image deployed on a Pod works fine.

Unknown User•2mo ago

Message Not Public

huwprosserOP•2mo ago

Thank you for your help! Unfortunately even deploying the example linked in that article lead to the same behaviour for me earlier today. I suppose I should probably open a ticket. Any further thoughts are welcome. 🙏

Unknown User•2mo ago

Message Not Public

huwprosserOP•2mo ago

It just hangs. I see the request in the metrics tab on the requests history as a failed request.

Unknown User•2mo ago

Message Not Public

flash-singh•2mo ago

default port is 80, use PORT env variable to specify any custom port

Unknown User•2mo ago

Message Not Public

flash-singh•2mo ago

whats the PORT env variable?

huwprosserOP•2mo ago

Good afternoon 🙏. Regardless of port variable we set we unfortunately see no improvement. Is it possible it HAS to be port 8888? - we haven’t tried that specific one yet. The /ping requests are however being logged by FastAPI.

flash-singh•2mo ago

ping your endpoint id, ill check logs just an update, the port is 80 by default, ignore the 8888 part

Unknown User•2mo ago

Message Not Public

flash-singh•2mo ago

that should work then

Unknown User•2mo ago

Message Not Public

flash-singh•2mo ago

how much slower is it compared to queue?

Unknown User•2mo ago

Message Not Public

flash-singh•2mo ago

your mainly talking about cold start being slower right?

Unknown User•2mo ago

Message Not Public

flash-singh•2mo ago

what does it do? just run the http server?

Unknown User•2mo ago

Message Not Public

flash-singh•2mo ago

interesting will raise this up, thats not ideal at all

Unknown User•2mo ago

Message Not Public

flash-singh•2mo ago

@Dj we need to raise this and also test to see if our examples are this slow

Dj•2mo ago

Looking into it, mostly just curious why Jasons took so long to wake up.

Unknown User•2mo ago

Message Not Public

Dj•2mo ago

These RT endpoints don't have the same logging for me, but I can imagine a part of the delay a lot of new Load Balancer users will see is from downloading a new Docker Image onto every server. I'll test the template we publish

flash-singh•2mo ago

An update to the original issue: Ignore the workers port 80 services blob and saying its not ready, we plan to remove that since it doesn't apply and creating extra confusion. The proper url to use for LB endpoints is to click Quick Start in overview tab and it will give you details on the url.

Dj•2mo ago

Every once in a while it takes an awkward amount of time to do... nothing.

❯ time curl -X POST "https://zvshz29pnyzitk.api.runpod.ai/generate" -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{"prompt": "Hello, world!"}'
{"generated_text":"Response to: Hello, world! (request #23)"} 0.409 total

❯ time curl -X POST "https://zvshz29pnyzitk.api.runpod.ai/generate" -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{"prompt": "Hello, world!"}'
{"generated_text":"Response to: Hello, world! (request #24)"} 5.376 total

❯ time curl -X POST "https://zvshz29pnyzitk.api.runpod.ai/generate" -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{"prompt": "Hello, world!"}'
{"generated_text":"Response to: Hello, world! (request #25)"} 3.662 total

❯ time curl -X POST "https://zvshz29pnyzitk.api.runpod.ai/generate" -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{"prompt": "Hello, world!"}'
{"generated_text":"Response to: Hello, world! (request #26)"} 0.310 total

❯ time curl -X POST "https://zvshz29pnyzitk.api.runpod.ai/generate" -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{"prompt": "Hello, world!"}'
{"generated_text":"Response to: Hello, world! (request #27)"} 0.390 total

❯ time curl -X POST "https://zvshz29pnyzitk.api.runpod.ai/generate" -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{"prompt": "Hello, world!"}'
{"generated_text":"Response to: Hello, world! (request #28)"} 0.295 total

❯ time curl -X POST "https://zvshz29pnyzitk.api.runpod.ai/generate" -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{"prompt": "Hello, world!"}'
{"generated_text":"Response to: Hello, world! (request #29)"} 0.307 total

❯ time curl -X POST "https://zvshz29pnyzitk.api.runpod.ai/generate" -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{"prompt": "Hello, world!"}'
{"generated_text":"Response to: Hello, world! (request #30)"} 0.308 total

❯ time curl -X POST "https://zvshz29pnyzitk.api.runpod.ai/generate" -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{"prompt": "Hello, world!"}'
{"generated_text":"Response to: Hello, world! (request #23)"} 0.409 total

❯ time curl -X POST "https://zvshz29pnyzitk.api.runpod.ai/generate" -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{"prompt": "Hello, world!"}'
{"generated_text":"Response to: Hello, world! (request #24)"} 5.376 total

❯ time curl -X POST "https://zvshz29pnyzitk.api.runpod.ai/generate" -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{"prompt": "Hello, world!"}'
{"generated_text":"Response to: Hello, world! (request #25)"} 3.662 total

❯ time curl -X POST "https://zvshz29pnyzitk.api.runpod.ai/generate" -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{"prompt": "Hello, world!"}'
{"generated_text":"Response to: Hello, world! (request #26)"} 0.310 total

❯ time curl -X POST "https://zvshz29pnyzitk.api.runpod.ai/generate" -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{"prompt": "Hello, world!"}'
{"generated_text":"Response to: Hello, world! (request #27)"} 0.390 total

❯ time curl -X POST "https://zvshz29pnyzitk.api.runpod.ai/generate" -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{"prompt": "Hello, world!"}'
{"generated_text":"Response to: Hello, world! (request #28)"} 0.295 total

❯ time curl -X POST "https://zvshz29pnyzitk.api.runpod.ai/generate" -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{"prompt": "Hello, world!"}'
{"generated_text":"Response to: Hello, world! (request #29)"} 0.307 total

❯ time curl -X POST "https://zvshz29pnyzitk.api.runpod.ai/generate" -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{"prompt": "Hello, world!"}'
{"generated_text":"Response to: Hello, world! (request #30)"} 0.308 total

But about 307ms is the average response time.

Dj•2mo ago

If I go to spam it, I can get pretty consistently low request times but if I pause to edit that output or write a message here consistently the same 1 worker goes back to 3-5 second response times.,