R
Railway

✋|help

Websocket disconnecting

Ffedev1008/26/2023
Hey, I have an app that uses a websocket (hosted on railway) but for some reason every once in a while (2 - 2.5h) the websocket just disconnectes (and then reconnects since i set up a reconnecting websocket) but since there is high traffic, in the second it disconnects i can loose a lot of data.

I saw another issue of this type and you guys suggested to use pings (which i'm sending like every 10 seconds both from the websocket and from the app connected to the websocket) and to use the .up.railway.app domain instead of a custom one as written here https://help.railway.app/troubleshooting/hKqw9Dr1moxfDySTy98E6G/websocket-connections-disconnecting/sb6bfjV6UnuMwkRyAvQ3Sb and i'm doing that too.

Ill also attach some logs of the client reconnecting to the websocket every 2 or 2.5h
Ffedev1008/26/2023
b16be651-2cf1-4464-91ef-3890dc9c63aa
(websocket service id)
Bbrody1928/26/2023
this is a websocket connection between a client and your railway service right?
Ffedev1008/26/2023
between a nodejs client and a nodejs websocket server (both on railway)
Bbrody1928/26/2023
are they in the same project?
Ffedev1008/26/2023
you mean service?
Bbrody1928/26/2023
I mean project
Ffedev1008/26/2023
no they're not
Bbrody1928/26/2023
should they be?
Ffedev1008/26/2023
hmm no
why?
Bbrody1928/26/2023
because it seems like they should, they talk to each other after all, and if you put them in the same project, you can use the internal networking
Ffedev1008/26/2023
but its a websocket server it doesent need to be in the same project, it works fine just that sometimes it disconnects
Bbrody1928/26/2023
also you miss read the help page, it says to use a custom domain instead of the railway domain, you have it the other way around
Ffedev1008/26/2023
oh god...😂
I feel so stupid ahah
Bbrody1928/26/2023
either way I'm kinda leaning towards this not being an issue with railway
your time between disconnects are not fixed times, some 2 hours, some 4 hours, etc
so I think you should start logging the disconnect reason, and once you have those error logs, go from there
Ffedev1008/26/2023
will try with custom domain and add some logs and see then 😅
Bbrody1928/26/2023
also also, I'm pretty sure fp or char told me that it has since been fixed a long time ago
Ffedev1008/26/2023
oh ok
Bbrody1928/26/2023
so yeah hold off on the custom domain for now
log both error and disconnect events on both ends of the websocket
Ffedev1008/26/2023
the only log i can get from websocket close event is the code
Ffedev1008/26/2023
Ffedev1008/26/2023
code 1006

 It is designated for use in
      applications expecting a status code to indicate that the
      connection was closed abnormally, e.g., without sending or
      receiving a Close control frame.
Bbrody1928/26/2023
there are error and close event emitters for the websocket connection, you will want to print the reason for error and the reason for disconnect respectively
Ffedev1008/26/2023
im listening to both of them
error is not firing, only close
Bbrody1928/26/2023
you will want to print the reason for error and the reason for disconnect respectively
Ffedev1008/26/2023
but how can i print the error if there is no error...
Bbrody1928/26/2023
if there was no error then you wouldn't have a disconnect, there is an error, print it
Ffedev1008/26/2023
    this.ws.on("close", closeMessage => {
      console.log("WEBSOCKET CLOSE", closeMessage); // 1006 WAS LOGGED
      if (this.shouldReconnect) {
        this.scheduleReconnect();
        this.stopPingTimer();
      }
    });

    this.ws.on("error", (error: any) => {
      // if (error.code === "ECONNREFUSED") return;
      console.log(`${this.nameIdentifier} WebSocket error`, error); // NOTHING WAS LOGGED
    });
Bbrody1928/26/2023
i just cant see how this issue is railways fault, given the fact that the time between disconnects is sporadic, if there where timeouts for websocket connections in place you would see constant time between disconnects
like i have said, log the reason, you are only logging the code
Ffedev1008/26/2023
ok then I will see if I can manage to figure it out thank you anyways

maybe the problem is that ws.send("ping") instead of ws.ping()
how am I not logging the reason ? there is console.log(error) and console.log(closeMessage)
Bbrody1928/26/2023
you are only logging the code
please read the reference docs for the close event
https://developer.mozilla.org/en-US/docs/Web/API/WebSocket/close_event
Ffedev1008/26/2023
you mean I'm doing console.log(closeMessage) instead of console.log(closeMessage.reason)?
Bbrody1928/26/2023
please read the reference docs for the close event
https://developer.mozilla.org/en-US/docs/Web/API/WebSocket/close_event
Ffedev1008/26/2023
I have read it
and still not understanding what you are saying
Bbrody1928/26/2023
I don't think my attempts to guide you in the right direction are bearing any fruit. I have other threads from other users to attend to and wish you the best as you debug your issue. Hopefully, it gets resolved! 🙂
Ffedev1008/26/2023
ok sorry, thank you anyway :railway:
Ffedev1008/27/2023
Hey brody, i'm pretty sure that the websocket disconnecting issue is a problem from railway

I have deployed the same websocket server on AWS and 11 hours later (for me it 20:10) the node client still hasen't disconnected from the websocket
Bbrody1928/27/2023
okay try a custom domain next
Ffedev1008/27/2023
Yes I already tried it yesterday
Bbrody1928/27/2023
try on fly?
Ffedev1008/27/2023
Wym?
Bbrody1928/27/2023
fly.io
you tried it on aws, now try it on fly.io
Ffedev1008/27/2023
Oh its a hosting platform?
Bbrody1928/27/2023
yeah
Ffedev1008/27/2023
Ok later will try to host on fly and tomorrow will see if it disconnected then
Ffedev1008/28/2023
I tried with fly and 10h and 30 min later its still connected
Ffedev1008/28/2023
i also retried with railway websocket again just to make sure and less than 2h later it disconnected
Bbrody1928/28/2023
okay, I'm gonna run a test myself too, if I can reproduce this, I will get the team involved for you
Ffedev1008/28/2023
Ok thanks, for the moment i deployed the websocket server on fly io
Bbrody1928/28/2023
what would you say the max time you have been able to keep a websocket connection open for on railway is?
Ffedev1008/28/2023
from this screen 9,5h but generally it's 2h
Bbrody1928/28/2023
well I hope I don't have to get back to you in 9.5 hours
I also kind hope I can reproduce this, I'm not sure what I'd say to you if I can't
Ffedev1008/28/2023
not urgent ahah
if you want i can provide you the code i'm using
Bbrody1928/28/2023
all we will be waiting for is my websockets test to disconnect
Ffedev1008/28/2023
yeah
Bbrody1928/28/2023
can reproduce
DDuchess8/28/2023
Thread has been flagged to Railway team by @Brody.
Bbrody1928/29/2023
@char8 - websocket disconnects
Cchar88/29/2023
Actively looking into this - have an idea of what it could be, so testing that hypothesis. Will update here when I have more.
Ffedev1009/8/2023
Hey any news?
Rrayofbytes9/8/2023
Hey, not yet sorry. It may be related to some other network issues we're going to investigate soon

The team's heads down on getting regions out atm

Looking for more? Join the community!

Recommended Posts
npm WARN config production Use `--omit=dev` instead when trying to start Nestjs appI'm trying to deploy my app (https://github.com/Omniforce-Technologies/omniforce-asset-store/tree/deDjangoHello good people of Railway! I would like to ask for a bit of help and advice about Django deploymeI have a custom domain from SuperHostingI have this doman briefeats.eu and I want to configure it to point to my angular app Currently I haDNS settings not being accepted by domain host.I'm finalizing a deployment of a React.js frontend only site, I got the green confirmation of the siWinston logs not appearing in deployment logsIn my node.js deployment if I do a console.log it appears normally in deployment logs, but using windeploy logs say nuxt "start" is an unknown command. Production and development sites both crashed.😫CMD Line accessI freaking swear I hate this so much I JUST WANT TO BE ABLE TO USE THE COMMAND LINE <:sadcat:7868689Laravel cron jobsHow can I run php artisan schedule:work every minute?Postgres drops connection when inactive, is this by default?I'm developing a node app using AdonisJS@5 and on a prolonged inactivity period (more than 5 hours) Fontconfig to canvasHi, I've been using the node-canvas in default railway when I use that, only libuuid and libGL are ivolumehey guys! quick question, I've got a golang project. I need it to create a json file on railway and system ran out of memoryCan someone help me? When I try to run my react app i get this error message. > tuya_react_frontend@Litestar command not foundHey all apologies for the possibly silly question. I am looking to deploy this repo: https://github.save temporary archivesCan i save temporary pdfs to a back-end host ? I'm trying to save a pdf file from a client upload bHow to host websockets?This is my first time hosting socket server, how can i do that on railway? I docs there are only webI cannot use opencv2 and received ImportError: libgl.so.1 cannot open shared object file no such filCan anyone help me with that ??LibreofficeHi I need libreoffice installed to use in my code. I attempted using the below nixpacks.toml ``` [Random app crash, no logs or other indicatorWhen trying my healtcheck I get random crashes, no recent deploy was made, I don't see any logs on mModify build commandHow would I modify the build command so it does a pip install of `psycopg2` ? I was told to remove tPayment method removalHey, how do I remove my payment method? I opted in for a credit subscriptions but still see my credi