Getting 503 Server Error during load tests

I'm load testing a simple fastapi backend running on railway. Using locust for load testing with 100 users with around 10 http requests per second. I observe that around 20% of http requests are returning 503 Server Error at locust client. Same setup works fine locally. Is railway throttling/limitting these requests or am I doing something wrong?

Any help will be appreciated. Thanks.

Sample Error:
POST /api/v1/auth/login: HTTPError('503 Server Error: Service Unavailable for url: /api/v1/auth/login')

Project Id: a5a6f516-4f3a-4f38-979b-b0689c7e5ab7

whats your start command on railway?

Didn't configure it, using the defaults

I don't think there is a default built into nixpacks for fastapi, would you happen to have a Procfile?

Nope, my assumption is nixpacks is picking up main.py from project root, here is how main.py looks like

import os
from app import app
import uvicorn


if name == "main":
uvicorn.run(app, host="0.0.0.0", port=int(os.getenv("PORT", 8000)))

yep that would do it, I was thinking about running the uvicorn command directly
either way, you may want to play around with more workers and more threads

Yep, have some plans for that, but for now I wanted to squeeze out performance from simple single worker config and see what it can deliver at its best ...

it looks like you've already done that

Maybe, but getting the 503 errors as mentioned above 😅

yeah seems like uvicorn can't keep up
I would be interested to see how hypercorn does

Not sure if uvicorn is having trouble with handling 10 requests per second, I seen it deliver much more. I'm looking more deep into the cause. However I strongly feel 503 Server Error type errors are something a proxy or a load balancer would throw to rate limit requests. Wanted to check if railway has some mechanism/service which is blocking these request by certain criteria or is it more to do with fastapi/uvicorn?

I'm positive railways envoy proxy can handle plenty more then 10 r/s
there would be no rate limiting, there's only metering

Thanks for the info, will now dig deeper into fastapi/uvicorn then ...

give hypercorn a try too!

Yep, on my todo list 🙂
So I carried few more experiments to find the root cause of 503 Server Error type errors:

Experiment 1: Original Setup (Default uvicorn config)
Result: I found following response header and text
{'content-type': 'text/html', 'x-railway-fallback': 'true', 'content-length': '2942', 'date': 'Mon, 04 Sep 2023 10:01:25 GMT', 'server': 'railway'}

Text: A html page which says
<h1 class="error-404">Nothing here... yet</h1>
<h1 class="error-503">Application failed to respond</h1>

<a href="https://railway.app" target="_blank"> Go to Railway </a>
</main>


Experiment 2: Changed uvicorn's limit_concurrency to 10 in order to force induce 503 errors.
Result: {'date': 'Mon, 04 Sep 2023 10:17:40 GMT', 'server': 'railway', 'content-type': 'text/plain; charset=utf-8', 'transfer-encoding': 'chunked'}
Service Unavailable

From above experiments I see a difference between response headers mainly 'x-railway-fallback': 'true' being set when the 503 error is not thrown directly by uvicorn/fastapi in Experiment 1, plus I don't see 503 errors in the railway's Observability section in the same Experiment.

Whereas 'x-railway-fallback': 'true' is missing from headers in Experiment 2 and I see 503 errors in the railway's Observability section in this experiment.

Any ideas what this could mean?
I did try hypercorn and it was quite effective in reducing error rate. However it does produce similar 503 Server Error type errors though in significantly less amount as compared to uvicorn.

interesting results, the endpoint you are stress testing, what does it return?

You are asking about expected success result or the erroneous result?

the expected response? like what does the endpoint you are testing return? or am I thinking about this incorrectly?

I'm stress testing complete user journey so I'm hitting around 13 endpoints which return response which vary in shape of empty response to a complex json response.

ah yes i see now