Railway•4mo ago

Server Response too slow

I have Django, React application live on railway. I'm using Hobby plan. whenever I click through pagination buttons server is taking up to a minute to respond. I can see in the network panel that request has been sent and it takes 10 - 60 second for the response to come in. on the localhost everything is instant. I don't know what's causing it. Full project code : https://github.com/AlexShonia/e-commerce pagination code on django:

@api_view(["GET"])
def getProducts(request):
    query = request.query_params.get("keyword")
    if query == "null" or query == None:
        query = ""

    products = Product.objects.filter(name__icontains=query)

    page = request.query_params.get("page")
    paginator = Paginator(products, 4)

    try:
        products = paginator.page(page)
    except PageNotAnInteger:
        products = paginator.page(1)
    except EmptyPage:
        products = paginator.page(paginator.num_pages)

    serializer = ProductSerializer(products, many=True)
    return Response(
        {"products": serializer.data, "page": page, "pages": paginator.num_pages}
    )

@api_view(["GET"])
def getProducts(request):
    query = request.query_params.get("keyword")
    if query == "null" or query == None:
        query = ""

    products = Product.objects.filter(name__icontains=query)

    page = request.query_params.get("page")
    paginator = Paginator(products, 4)

    try:
        products = paginator.page(page)
    except PageNotAnInteger:
        products = paginator.page(1)
    except EmptyPage:
        products = paginator.page(paginator.num_pages)

    serializer = ProductSerializer(products, many=True)
    return Response(
        {"products": serializer.data, "page": page, "pages": paginator.num_pages}
    )

GitHub

GitHub - AlexShonia/e-commerce

Contribute to AlexShonia/e-commerce development by creating an account on GitHub.

121 Replies

Percy•4mo ago

Project ID: 59308ce1-f27b-4a5b-aa0b-10ac909d0bba

AshoniA•4mo ago

59308ce1-f27b-4a5b-aa0b-10ac909d0bba

Brody•4mo ago

how are you serving your frontend?

AshoniA•4mo ago

npm run build and python manage.py collectstatic

Brody•4mo ago

https://github.com/AlexShonia/e-commerce/blob/main/backend/backend/settings.py#L107C9-L107C59 this is never a good idea please separate this repo out into two folders -

backend
  manage.py
  ...
frontend
  package.json
  ...

backend
  manage.py
  ...
frontend
  package.json
  ...

AshoniA•4mo ago

if I did this I'm guessing I'd have to change how I'm specifying staticfiles path here: https://github.com/AlexShonia/e-commerce/blob/174204655011dc89bf06e2f3f34858ea2f5576e7/backend/backend/settings.py#L169 would there be anything else to change?

Brody•4mo ago

there is many things to change, serving the frontend from a django backend is far from ideal

AshoniA•4mo ago

if I didn't serve frontend from django backend would that mean I'd have to host frontend separately from backend? I'm quite new to this so I don't know the best practices. where would I find more information about this?

Brody•4mo ago

yes your frontend will be served by another railway service, that is standard practice finding more information about this means just following along while I guide you

AshoniA•4mo ago

Alright I'll start seperating the two

Brody•4mo ago

awesome, that means also removing anything from the backend code that is meant to serve the frontend

AshoniA•4mo ago

Alright, I'll try to do that I'm guessing staticfiles directory and the code connected to it will be removed on the backend

Brody•4mo ago

yeah all the frontend assets will be served by the frontend service

AshoniA•4mo ago

what about images that users can upload? can backend store them in the public folder on the frontend?

Brody•4mo ago

yeah that's backend stuff

AshoniA•4mo ago

I see idk what happened, but my github is not available anymore Account was suspended

Brody•4mo ago

well that's not ideal

AshoniA•4mo ago

so weird wow yeah not really lmao it almost feels like you hacked me but that's just the coincidence :D

Brody•4mo ago

haha

AshoniA•4mo ago

this is basically my portfolio website and maybe they thought it was actualy e-comemrce website and I didnt meet rules or something is probably what happened oh well will talk to support

Brody•4mo ago

sounds good, feel free to ping me when you have that sorted!

AshoniA•4mo ago

Alright, thanks for the help so far!

Brody•4mo ago

no prob

AshoniA•4mo ago

Okay I'm back. idk what that was but after some time I just gained access again. weird. anyway I spearated the repositories and deleted static files and some code connected to it https://github.com/AlexShonia/e-commerce/commits/main/

Brody•4mo ago

please remove all code from the backend that serves the frontend like this - https://github.com/AlexShonia/e-commerce/blob/main/backend/backend/settings.py#L107C9-L107C59

AshoniA•4mo ago

did it

Brody•4mo ago

make sure you remove the frontend's dist folder from github and add it to your .gitignore

AshoniA•4mo ago

done

AshoniA•4mo ago

AshoniA•4mo ago

alright uploaded this repo twice in one settings I have backend/ as root and in another frontend/ I'm guessing this is how I'd deploy front and back from same repository seperately

Brody•4mo ago

well hold on there can i ask you to switch to postgres instead of sqlite?

AshoniA•4mo ago

sure that would be better

Brody•4mo ago

yes it really would

AshoniA•4mo ago

okay that should be it, I made a postgres database on railway and connected it with settings.py:

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": os.environ.get("PGDATABASE"),
        "USER" : os.environ.get("PGUSER"),
        "PASSWORD" : os.environ.get("PGPASSWORD"),
        "HOST" : os.environ.get("PGHOST"),
        "PORT" : os.environ.get("PGPORT"),
    }
}

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": os.environ.get("PGDATABASE"),
        "USER" : os.environ.get("PGUSER"),
        "PASSWORD" : os.environ.get("PGPASSWORD"),
        "HOST" : os.environ.get("PGHOST"),
        "PORT" : os.environ.get("PGPORT"),
    }
}

Brody•4mo ago

looks good to me, do you have all those variables set in your service variables for the backend service?

AshoniA•4mo ago

not on backend, they are set on postgres

AshoniA•4mo ago

should I set them there too? alright I set them on backend service as well

Brody•4mo ago

click the eye icon on the backend's service variables to make sure they show correctly

AshoniA•4mo ago

oh yeah some are empty for some reason even though I copied raw text I'll fix that. also Im getting this errors on frontend i the logs. I'll look into that

AshoniA•4mo ago

so, I'm changing these variables and still they show up as empty strings

AshoniA•4mo ago

when I click edit they appear again maybe just a visual bug

Brody•4mo ago

please slow down, there are changes we need to make still, you wont be able to just give railway your repo just yet these are what you want to use for your app's database variables, not what you currently have

PGDATABASE=${{Postgres.PGDATABASE}}
PGUSER=${{Postgres.PGUSER}}
PGPASSWORD=${{Postgres.PGPASSWORD}}
PGHOST=${{Postgres.PGHOST}}
PGPORT=${{Postgres.PGPORT}}

PGDATABASE=${{Postgres.PGDATABASE}}
PGUSER=${{Postgres.PGUSER}}
PGPASSWORD=${{Postgres.PGPASSWORD}}
PGHOST=${{Postgres.PGHOST}}
PGPORT=${{Postgres.PGPORT}}

AshoniA•4mo ago

sorry, I'm just trying to fix stuff on my own. I'll wait for you then alright these are set now on my backend service

AshoniA•4mo ago

oh nvm they are differnt

Brody•4mo ago

please use the variables i provided

AshoniA•4mo ago

yep doing it now

AshoniA•4mo ago

alright should be it

Brody•4mo ago

is this the raw editor of the postgres service?

AshoniA•4mo ago

it's of the backend service

Brody•4mo ago

why do you have every variable in there that the postgres service has?

AshoniA•4mo ago

I copied everything from there😅 so should I just use only the ones you provided?

Brody•4mo ago

yes

AshoniA•4mo ago

Brody•4mo ago

okay now make sure they display correctly

AshoniA•4mo ago

pghost and pgport are empty strings still

AshoniA•4mo ago

others appear fine

Brody•4mo ago

you would need to save the changes with alt + shift + enter

AshoniA•4mo ago

alright, eveyrhting appears now

Brody•4mo ago

okay can i see a screenshot of your railway project now please

AshoniA•4mo ago

if this is what you meant, here it is

Brody•4mo ago

yeah thats good okay code changes time

AshoniA•4mo ago

im ready :)

AshoniA•4mo ago

haha got my answer on that one

Brody•4mo ago

lmao in your backend, delete the Procfile and replace it with this railway.json file

{
    "$schema": "https://railway.app/railway.schema.json",
    "build": {
        "builder": "NIXPACKS"
    },
    "deploy": {
        "startCommand": "python manage.py migrate && gunicorn backend.wsgi --log-file -",
        "restartPolicyType": "ON_FAILURE",
        "restartPolicyMaxRetries": 10
    }
}

{
    "$schema": "https://railway.app/railway.schema.json",
    "build": {
        "builder": "NIXPACKS"
    },
    "deploy": {
        "startCommand": "python manage.py migrate && gunicorn backend.wsgi --log-file -",
        "restartPolicyType": "ON_FAILURE",
        "restartPolicyMaxRetries": 10
    }
}

and the root directory should be set to /backend

AshoniA•4mo ago

did it

Brody•4mo ago

okay now let me know if the backend fails

AshoniA•4mo ago

it's successful

AshoniA•4mo ago

it crashed

SECRET_KEY = os.environ.get("SECRET_KEY")

SECRET_KEY = os.environ.get("SECRET_KEY")

I get secret key from environment variables and I dont have it set anymore guess I'll just set it again

Brody•4mo ago

yeah just set it as a service variable

AshoniA•4mo ago

alright it's up

AshoniA•4mo ago

these is the log output though

Brody•4mo ago

ive updated this

AshoniA•4mo ago

alright it's active now

Brody•4mo ago

awsome

AshoniA•4mo ago

nice

Brody•4mo ago

on to the frontend

AshoniA•4mo ago

let's go

Brody•4mo ago

https://github.com/brody192/vite-react-template copy the nixpacks.toml and Caddyfile from this repo into your frontend folder. you will want the root directory on the frontend service set to /frontend

AshoniA•4mo ago

alright did it

Brody•4mo ago

and let me know if something with the frontend fails

AshoniA•4mo ago

service is active these are the logs

Brody•4mo ago

okay now you will have to update your frontend code to call the backend domain

AshoniA•4mo ago

alright I think I know what to do

Brody•4mo ago

awsome

AshoniA•4mo ago

alright the server is working! you're awesome man, can't thank you enough

Brody•4mo ago

nothing is too slow?

AshoniA•4mo ago

have to test it out yet after I add stuff to db I'll let you know !

Brody•4mo ago

sounds good

AshoniA•4mo ago

hmm the problem remains still, must be something with my code then. regardless it was great to improve the architecture and conect to postgres

Brody•4mo ago

do you have some middleware that's slowing things down?

AshoniA•4mo ago

not that I know of. only the pagination is the problem it seems. registration and login is pretty fast as well as other operations

Brody•4mo ago

well to be honest, I wasn't fully expecting these changes to fix anything but it at least does mean it's an issue with the code and not the structure, so it was definitely worth while to do this

AshoniA•4mo ago

definitely worth a while, and thank you again!

Brody•4mo ago

happy to help where I can, and definitely let me know when you find out what code is responsible for the slow down

AshoniA•4mo ago

I'll let you know! I think I found out what the problem is lol. I just found out the Server is deplyed in US West Oregon and I'm in Georgia (country) so the ping should be very high. I wonder how the site is doing in US.

Brody•4mo ago

how long is too slow again? you said 60 seconds right?

AshoniA•4mo ago

sometime it's quite fast. 1-3 seconds. and sometimes it's like 30 seconds and worker times out. around 2 out of 5 times the worker times out, and thats a lot

Brody•4mo ago

is there a page link you could send me so i could test? im not us-west but i am far closer than you are

AshoniA•4mo ago

when I'm in localhost and I make connection to local postgres db the responses are very fast (under a second) so thats why I'm thinking it's not the code but where the site is hosted. https://e-commerce-sportshop.up.railway.app/ sure, just pagination from page to page is the problematic part other parts too but this is most problematic

Brody•4mo ago

just loading the content for that page took ~17 seconds, distance is not the main issue you are facing

AshoniA•4mo ago

damn, okay. in one way it's good news and means I can fix this

Brody•4mo ago

indeed

AshoniA•4mo ago

from what I've tested the problematic line is this in the django:

    if(query):
        products = Product.objects.filter(name__icontains=query).order_by("_id")
    else:
        products = Product.objects.all().order_by("_id")

    if(query):
        products = Product.objects.filter(name__icontains=query).order_by("_id")
    else:
        products = Product.objects.all().order_by("_id")

so querying data from db is the problem I tested this by printing something first and then printing products and products got printed much later

Brody•4mo ago

interesting

AshoniA•4mo ago

After countless hours of researching I found out that the problem appears only on firefox. I tested edge and chrome and they are all fine. yes the response sometimes takes 1-3 seconds but it's only expected as server is so far away. and nothing compared to 30+ seconds. and btw it's not actually 30+ seconds it's just that gunicorn times out after 30 seconds and when I set timeout to 5 minutes the problematic responses took all 5minutes. now this problem doesn't bother me as much now because most people use chrome anyway and this is just a portfolio website. but another problem I found is that backend django is in debug mode still even though I specified Debug= False. wait nvm, I thought debug mode meant that I wouldn't even be able to access /admin page but I can't login there so I guess debug is indeed false

Brody•4mo ago

I was on chrome

AshoniA•4mo ago

that was a different problem. I think I fixed that by usign collectstatic in start configurations? idk or maybe because I changed debug to false. but the 30+ sec response doesn't happen on chrome. and the load times are good for me

Brody•4mo ago

load times are good for me now too

AshoniA•4mo ago

nice

Brody•4mo ago

job done

webomania•3mo ago

I have the same problem, and I am on chrome

Brody•3mo ago

please open your own help thread

Gaming

Programming

Server Response too slow