Server becomes unresponsive

Can someone please help me figure out what occurred Saturday? Our production site was not accessible during a live event. We were able to reach the front end, but the data wasn’t loading. We are using express JS with node and a MYSQL database.

Both the back-end and front-end servers are hosted on Railway.app The problem was fixed by redeploying the backend server. The problem has occurred again several times over the past few days, every time it is fixed by redeploying the backend server.


During each outage, we have been able to run queries directly on the production database with MySQL workbench. It works like normal.

The SQL command:
SHOW VARIABLES LIKE "max_connections";
returned 151.

The SQL command:
SHOW VARIABLES LIKE "max_used_connections";
returned nothing.
I do not think it is related to the database.

We have a cron job (Node Cron) that runs every minute, it stops running during the outages. We have console.logs in the cron job allowing us to know when it begins and ends. The cron jobs always finish.
I do not think it is related the the cron job.

We use the Morgan logger, which displays the routes we hit in the logs. When going through the logs, we can see when the outage begins because we stop getting status and response time.

Morgan is logging:
:method :url :status :response-time ms - :res[content-length]
Sample responses before an outage begins:
GET /api/bouts/allBoutsForAnEvent/361 200 2.567 ms - 2
GET /api/registrations/getAllRegisteredCompetitorsForSingleEvent/361 200 6.167 ms - 34436
OPTIONS /api/users/getUserRoleFromToken 200 0.129 ms - 0
OPTIONS /api/wrestlers/getWrestlerRankings/62877 200 0.162 ms - 0
GET /api/wrestlers/getWrestlerProfile/62877 200 4.315 ms - 323
GET /api/wrestlers/getWrestlerWeighInForProfile/62877 200 4.828 ms - 155
GET /api/WARZone/getWrestlingStyles 200 5.608 ms - 241

Sample responses during an outage:
GET /api/events/getEventForCustomerView/367 - - ms - -
GET /api/events/getEventForCustomerView/361 - - ms - -
PUT /api/events/allEventsForCustomerHomePage - - ms - -
GET /api/events/getEventForCustomerView/366 - - ms - -
OPTIONS /api/users/getSingleUser 200 1.465 ms - 0
OPTIONS /api/events/allEventsForCustomerHomePage 200 0.087 ms - 0
GET /api/production/backgroundColorForDevelopmentOrTesting 200 0.192 ms - -
POST /api/users/getSingleUser 500 0.534 ms - 148

I hit the API during an outage with Postman, which results in “stream timeout” after about 5 minutes. There is a large delay from when I send the postman request to when Morgan logs it, as there are many other requests that continue to come in. I shut down the front-end server (so that I know additional requests cannot be coming from anywhere other than Postman), but the requests continued logging for several minutes. This leads me to believe we have a backlog of requests building up.
my project ID is: 1fc9ac5f-29f8-4b23-ab28-1eebdaeb8c06
At this point I believe we are being DDOS'd, we're getting an unusual amount of requests to /auth/login
I don't know what to do, does railway provide any DDOS protection?
railway has not had any reported networking or host outages, so im sorry to say but at this point it looks a lot like your app is soft locking, this would be a problem with your code and not the platform, if you are being ddosed in some capacity and your app cant handling the requests you will want to use cloudflare maybe with some specific stricter blocking rules in place, since railway does not provide any ddos protection
Ok, thank you Brody
sorry we cant be of much more help than that

Looking for more? Join the community!

Recommended Posts
Project with cron job aways skippedWhy my project nevers run with cronMaximum of 4 volumes per project?Heya 🌞 I have several services and would need to attach another volume to one of these but it get aAny way to skip redeployment if github ci is skipped?Here's what I see on dashboard, railway is granted all accesses in my org on github. On github I caBuild Error after RedeployRecently, I've been facing a weird error whenever I redeploy my railway instance from the dashboard.Restarting database not working.My postgres database wont restart. All data is not modifiy after restaring it.Adding support for python with isolated-vm npm packageHello, I am using a project that requires the isolated-vm npm package (https://www.npmjs.com/packageListmonk; Static FilesNot sure why my post was deleted, but here it is again: Hello, I've deployed Listmonk on Railway, ConcurrencyWill concurrent requests to a single endpoint work? And where would the logs appear for each individProblem with Flask-Caching/ RedisI'm trying to setup a simple caching mecanism to my Plotly Dash (Flask) application deployed to railWorker Killed IssueI'm getting the attached error when I try to request my app from a React frontend, whereas I wasn't Need help with deploymentSubject: Seeking Guidance on Loading Environment Variables in Railway for My Go API Deployment Hi ebun: command not foundHello, I'm trying to migrate my project to bun to make TS easier, and I'm when deploying, the logs Metrics on Number of Web RequestsI just deployed a web API and would like to keep track of the number of requests made to the API. DoIs there a timeout on my api functions when deploying a nextjs app?I coudnt find anything about this on google or the docs, Thanks for helpingI have problems with the database when I do a deploy.I have a mongo DB database that is connected to a node js, to connect to mongo I use the url that moI have problems with the database when I do a deploy.I have a mongo DB database that is connected to a node js, to connect to mongo I use the url that moError comunicate with internal servicecsI'm trying to make a post on a route that is in an internal service on railway = api-ecompleto.railwUnchanged repo fails to deploy on env var change.I had this exact setup running last night, and this morning when I changed an environment variable iIs it possible to host Unity project?I have a unity game development project which runs C# Since I already use railway for my other projCan't see organizations reposTitle.