Railway Private Networking
This morning I checked my usage and saw a large jump in the estimated monthly cost (egress) for one of my projects and I'm looking to update so that the network communication between my stuff is over the private network.
I have a SvelteKit web UI that I'm updating to make API calls to my Golang REST API on the server-side using the provided private
*.railway.internal URL, however, the requests are failing because both port 443 and 80 are ERRCONNREFUSED, so I'd like help with that.
Also, the Golang REST API connects to a PostgreSQL (TimescaleDB) instance, and I would like some help figuring out how to connect from my REST API to the database over private network as well!
Cheers! 😃58 Replies
Project ID:
81d08146-e275-403d-8a25-046edbf806f681d08146-e275-403d-8a25-046edbf806f6
Do I not need to provide a scheme (http or https) when calling my backend API? I'm going to try that now
Solution
you need to use http, and specify the port
Also, the Golang REST API connects to a PostgreSQL (TimescaleDB) instance, and I would like some help figuring out how to connect from my REST API to the database over private network as well!going to writing a little guide on this
so
http://example.railway.internal:80/api/path ?what port does your go app listen on?
well crap lol, it looks like it's probably using
8000 as a defaultthen use 8000 in that url
this is the private network, there is no proxy and no firewalls, anything a service listens on is exposed within the private network
tcp, udp, http, anything, you just need to use the correct scheme, and port when connecting
okay so it turns out that railway was automatically setting a port for my backend dynamically and I've updated my api to listen on 80.
GitHub is having some issues right now so my changes aren't rolling out but that should fix the UI -> API communication over the internal network, next is the database which I'll handle once there's a guide for it 🙂
haha you could have just set a fixed PORT variable
yep exactly, and that's what I just ended up doing 😅
you should use a non privileged port like 8080
still working on that guide though
I just noticed the PG over Private Network guide in a separate thread, and I'm not sure I'll be able to utilize that until I migrate away from using the Postgres Plugin Database and onto the volume-based database
you are correct
something I've been putting off for weeks but it looks like I'll have to make some time for this very soon
railway is soon coming out with a one click migrate button
😮
oh man that's awesome, I can probably wait until that's available cause plugins are sunsetting soon right?
or you can do it now semi-manually with https://railway.app/template/postgres-plugin-migration
did the guide help? 🙂
I'll get back to you once I've used it lol - I'm currently debating on if when I want to move ahead with the plugin migration 😅
I know I really should get this taken care of as soon as possible, I just have other things I'm focusing on at the moment
But the private network between my UI & REST API already basically halved my egress, so I'm happy for now
awsome, let me know if you run into any issues during the migration
I did run into an issue using the migration template:
From this here in the script - https://github.com/railwayapp-templates/postgres-plugin-migration/blob/main/migrate.sh#L114-L115
The new Postgres DB isn't a timescale image, so it should probably skip that line?
yeah ill be letting jr know this
i'll open a PR that adds a condition check for the table, but maybe it's best to just remove those lines?
Is TimescaleDB even an option in the new volume-based databases?
railway provides a TimescaleDB template, its just not the default
https://railway.app/template/VSbF5V
there were also some of these warning showing up on the dump
i'll skip doing a PR, I don't really have a great way to test out any changes 😅
it can wait
I'm currently running the Postgres Plugin Migration to migrate my plugin DB to a new volume-based Postgres DB - The migration is running ( I think ) but the plugin migration service is regularly capped at 32GB Memory - I suspect this is the sed -i commands commenting out the timescaledb plugin creation?
I haven't seen any data come into the new DB yet, so maybe I'll just let this run for a bit? I'm just worried about it being steadily at 32GB Memory
would you say you have a lot of data in your database?
the database itself is ~11GB - but the log line containing the dump says its 30G, so that's what leads me to suspect the
sed -i commands?yeah guess sed is editing the file in memory?
@jr - enormous memory usage from migration service
hmm yes that is a problem. Taking a look
Feel free to delete the migration service if it is taking way too long. I will report back once that has been patched
I'm going let it keep going just in case there's any other issues
youre charged minutely usage for memory, id recommend closing it as it has already likely locked up
okay, good call - i'll stop it
here's what I was seeing, it looks like it took some time to finish but these are what I believe are the
sed -i spikes?
i wouldnt trust the data on the new database if i where you
these are the metrics of the plugin migration service, i canceled it and my target DB is still empty
and by "took some time to finish" - i mean the sed commands
It doesn't look like there was any data transfered. It was the
sed commands. I'm working on replacing those with something more efficient to handle massive files
I've replaced sed with awk in the migration script which should be more memory efficient. I'm going to add some more optimizations, but it shouldn't jump to 32gb of memory. It may still take a bit to process the large database thoughOkay, giving it a fresh run now! 😃
Looks like it was successful!
You may want to restart the new database to clear anything in memory after the restore
yep, i'll give it a restart, just checking the data is all good locally real quick 🙂
oh yeah for sure. Definitely check everything is all good and share if anything is off
looks good! now i just gotta take care of my application cutover
now you can use the private url!
Keep in mind any data written to the plugin after the migration will not be in the new database. You are also free to keep the legacy plugin around for a bit after cutting over so that you can be very confident everything is there
after changing the env variables with the private networking info my Go server is having trouble connecting to the database
I'm using the gorm driver, which takes a DSN formatted string
I think this may be the network initialization time. Can you try adding a small delay (500ms-1s) before the server starts? https://docs.railway.app/reference/private-networking#caveats
If it is still happening, it is something else. But we can debug it
i upped it to a 10s sleep and that seemed to give it enough time
still encountered the error on a 1s sleep
ah okay. 10s is a lot and 2s should be okay. You can also add a healthcheck to enusre that your application is up and ready before the old deployment goes down
you should only need a max of 3 seconds, the highest time ive ever seen it take was 3.67s
This is also something we are actively working on. Literally a single engineer is dedicated to improving this part of Railway
databases as services are cool, but having no initialization time is cooler
that's good stuff 🙂 I'll definitely need to start making use of healthchecks going forward
I think I'm okay with leaving the startup sleep at 10s for now - as far as I know nobody really visits this website haha - it really just exists for me at this point
to experiment with stuff
the delay would only effect the initial app start time
yep, and the only time it really starts up is if I'm pushing changes - I've never really had it crash or restart on it's own, it runs pretty smoothly
for sure still use a healthcheck, even just a simple one that returns a 200 status code
Hey everyone. Is there a way to accept UDP traffic from public network? and is it possible to expose more than one port?
unfortunately
1. nope
2. nope