Runpod•9mo ago

NGINX, Uvicorn and FastAPI setup not working

I'm going to put as much information here because I'm so lost and hopefully it makes helping me easier.

I've got an uvicorn server running and have verified it works when I ssh into the pod.

INFO:     Started server process [748]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

INFO:     Started server process [748]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Here's my nginx config

server {
    listen 8888;
    server_name 213.144.200.206;
    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

server {
    listen 8888;
    server_name 213.144.200.206;
    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

I've put that into /etc/nginx/sites-available/ and made a symlink to /etc/nginx/sites-enabled/

netstat gives this

ss -tuln | grep :8000
tcp   LISTEN 0      2048         0.0.0.0:8000       0.0.0.0:*          
root@8eccca76db93:/workspace# ss -tuln | grep :8888

ss -tuln | grep :8000
tcp   LISTEN 0      2048         0.0.0.0:8000       0.0.0.0:*          
root@8eccca76db93:/workspace# ss -tuln | grep :8888

So I can see the uvicorn server is listening, but nginx isn't listening on 8888

I've configured my pod to expose tcp port 8888

RiceOP•4/2/25, 7:36 PM

also I run nginx -s reload before starting my uvicorn server

RiceOP•4/3/25, 12:48 PM

RiceOP•4/3/25, 12:49 PM

I'm not sure how to confirm it's running

RiceOP•4/3/25, 12:49 PM

but here's some command output and it still doesn't listen to the port i specified

riverfog7•4/3/25, 12:53 PM

maybe looking at the logs from the service will help

RiceOP•4/3/25, 12:57 PM

how can i do that?

RiceOP•4/3/25, 1:00 PM

cat /var/log/nginx/error.log
2025/04/02 19:12:36 [notice] 440#440: signal process started
2025/04/02 19:25:16 [notice] 747#747: signal process started
2025/04/02 19:33:27 [notice] 964#964: signal process started
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:9091 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:3001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7861 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8081 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7270 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:9091 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:3001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7861 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8081 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7270 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:9091 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:3001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7861 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8081 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7270 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:9091 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:3001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7861 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8081 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7270 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:9091 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:3001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7861 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8081 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7270 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: still could not bind()
2025/04/03 12:47:04 [notice] 1491#1491: signal process started
2025/04/03 12:50:46 [notice] 1672#1672: signal process started
2025/04/03 12:56:26 [notice] 1786#1786: signal process started
2025/04/03 12:58:33 [notice] 1933#1933: signal process started

RiceOP•4/3/25, 1:00 PM

here's the nginx error log

riverfog7•4/3/25, 1:09 PM

why does it try to bind to random adresses

RiceOP•4/3/25, 1:11 PM

not sure?

riverfog7•4/3/25, 1:13 PM

yes

RiceOP•4/3/25, 1:14 PM

i don't know but my config i added should just do 8888

riverfog7•4/3/25, 1:17 PM

maybe because it is in a docker container?

riverfog7•4/3/25, 1:17 PM

chat gpt says try this
listen 0.0.0.0:8888;

RiceOP•4/3/25, 1:20 PM

doesn't seem to work, nothing changed with netstat or curl

riverfog7•4/3/25, 1:21 PM

aren't there any other configs

riverfog7•4/3/25, 1:22 PM

its really strange. why is it trying those ports

RiceOP•4/3/25, 1:22 PM

the runpod came with a default config

riverfog7•4/3/25, 1:22 PM

oh then

riverfog7•4/3/25, 1:22 PM

maybe nginx is already running

riverfog7•4/3/25, 1:22 PM

and you are trying to run 1 more

Rriverfog7 maybe nginx is already running

riverfog7•4/3/25, 1:23 PM

i mean not with services, in the docker CMD (or start command)

riverfog7•4/3/25, 1:23 PM

can you specify the image

RiceOP•4/3/25, 1:23 PM

maybe? im using the default pytorch 2.4 docker image

riverfog7•4/3/25, 1:24 PM

wait a sec

RiceOP•4/3/25, 1:25 PM

cat /etc/nginx/sites-available/default
##

You should look at the following URL's in order to grasp a solid understanding

of Nginx configuration files in order to fully unleash the power of Nginx.

https://www.nginx.com/resources/wiki/start/

https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/

https://wiki.debian.org/Nginx/DirectoryStructure

In most cases, administrators will remove this file from sites-enabled/ and

leave it as reference inside of sites-available where it will continue to be

updated by the nginx packaging team.

This file will automatically load configuration files provided by other

applications, such as Drupal or Wordpress. These applications will be made

available underneath a path with that package name, such as /drupal8.

Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.

Default server configuration

#
server {
listen 80 default_server;
listen [::]:80 default_server;

# SSL configuration
#
# listen 443 ssl default_server;
# listen [::]:443 ssl default_server;
#
# Note: You should disable gzip for SSL traffic.
# See: https://bugs.debian.org/773332
#
# Read up on ssl_ciphers to ensure a secure configuration.
# See: https://bugs.debian.org/765782
#
# Self signed certs generated by the ssl-cert package
# Don't use them in a production server!
#
# include snippets/snakeoil.conf;

root /var/www/html;

# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;

servername ;

location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}

# pass PHP scripts to FastCGI server
#
#location ~ .php$ {
# include snippets/fastcgi-php.conf;
#
# # With php-fpm (or other unix sockets):
# fastcgi_pass unix:/run/php/php7.4-fpm.sock;
# # With php-cgi (or other tcp sockets):
# fastcgi_pass 127.0.0.1:9000;
#}

# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /.ht {
# deny all;
#}
}

NGINX Product Documentation

Learn how to deliver, manage, and protect your applications using F5 NGINX products.

NGINX Product Documentation

Learn how to deliver, manage, and protect your applications using F5 NGINX products.

RiceOP•4/3/25, 1:25 PM

here's the default nginx config

RiceOP•4/3/25, 1:25 PM

in the site-available folder i got llm_app that i added and default that came with the docker image

riverfog7•4/3/25, 1:27 PM

riverfog7•4/3/25, 1:27 PM

can you cat the /start.sh

RiceOP•4/3/25, 1:28 PM

cat /start.sh
#!/bin/bash
set -e # Exit the script if any statement returns a non-true return value

---------------------------------------------------------------------------- #

Function Definitions #

---------------------------------------------------------------------------- #

Start nginx service

start_nginx() {
echo "Starting Nginx service..."
service nginx start
}

Execute script if exists

execute_script() {
local script_path=$1
local script_msg=$2
if [[ -f ${script_path} ]]; then
echo "${script_msg}"
bash ${script_path}
fi
}

Setup ssh

setup_ssh() {
if [[ $PUBLIC_KEY ]]; then
echo "Setting up SSH..."
mkdir -p ~/.ssh
echo "$PUBLIC_KEY" >> ~/.ssh/authorized_keys
chmod 700 -R ~/.ssh

if [ ! -f /etc/ssh/ssh_host_rsa_key ]; then
ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -q -N ''
echo "RSA key fingerprint:"
ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub
fi

if [ ! -f /etc/ssh/ssh_host_dsa_key ]; then
ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -q -N ''
echo "DSA key fingerprint:"
ssh-keygen -lf /etc/ssh/ssh_host_dsa_key.pub
fi

if [ ! -f /etc/ssh/ssh_host_ecdsa_key ]; then
ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -q -N ''
echo "ECDSA key fingerprint:"
ssh-keygen -lf /etc/ssh/ssh_host_ecdsa_key.pub
fi

if [ ! -f /etc/ssh/ssh_host_ed25519_key ]; then
ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -q -N ''
echo "ED25519 key fingerprint:"
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
fi

service ssh start

echo "SSH host keys:"
for key in /etc/ssh/.pub; do
echo "Key: $key"
ssh-keygen -lf $key
done
fi
}

Export env vars
export_envvars() {
echo "Exporting environment variables..."
printenv | grep -E '^RUNPOD|^PATH=|^_=' | awk -F = '{ print "export " $1 "="" $2 """ }' >> /etc/rp_environment
echo 'source /etc/rp_environment' >> ~/.bashrc
}

Start jupyter lab
start_jupyter() {
if [[ $JUPYTER_PASSWORD ]]; then
echo "Starting Jupyter Lab..."
mkdir -p /workspace && \
cd / && \
nohup jupyter lab --allow-root --no-browser --port=8888 --ip= --FileContentsManager.delete_to_trash=False --ServerApp.terminado_settings='{"shell_command":["/bin/bash"]}' --ServerApp.token=$JUPYTER_PASSWORD --ServerApp.allow_origin=* --ServerApp.preferred_dir=/workspace &> /jupyter.log &
echo "Jupyter Lab started"
fi
}

---------------------------------------------------------------------------- #

Main Program #

---------------------------------------------------------------------------- #

start_nginx

execute_script "/pre_start.sh" "Running pre-start script..."

echo "Pod Started"

setup_ssh
start_jupyter
export_env_vars

execute_script "/post_start.sh" "Running post-start script..."

echo "Start script(s) finished, pod is ready to use."

sleep infinity

RiceOP•4/3/25, 1:28 PM

here you go

riverfog7•4/3/25, 1:29 PM

service nginx restart

riverfog7•4/3/25, 1:30 PM

and

riverfog7•4/3/25, 1:30 PM

service nginx status

riverfog7•4/3/25, 1:30 PM

and it should work

riverfog7•4/3/25, 1:31 PM

apparently nginx was running on the default template so you got the port conflicts

RiceOP•4/3/25, 1:31 PM

huh

riverfog7•4/3/25, 1:31 PM

look at the start of the /start.sh file

riverfog7•4/3/25, 1:32 PM

there's this

riverfog7•4/3/25, 1:32 PM

and then

riverfog7•4/3/25, 1:32 PM

see that ?

RiceOP•4/3/25, 1:32 PM

yh?

riverfog7•4/3/25, 1:32 PM

so does it work?

RiceOP•4/3/25, 1:33 PM

ok its printing starting nginx nginx and nginx is running that follows

riverfog7•4/3/25, 1:34 PM

how about the ports?