R
Railway•13mo ago
CoCa

Railway Private Networking

This morning I checked my usage and saw a large jump in the estimated monthly cost (egress) for one of my projects and I'm looking to update so that the network communication between my stuff is over the private network. I have a SvelteKit web UI that I'm updating to make API calls to my Golang REST API on the server-side using the provided private *.railway.internal URL, however, the requests are failing because both port 443 and 80 are ERRCONNREFUSED, so I'd like help with that. Also, the Golang REST API connects to a PostgreSQL (TimescaleDB) instance, and I would like some help figuring out how to connect from my REST API to the database over private network as well! Cheers! 😃
Solution:
you need to use http, and specify the port
Jump to solution
58 Replies
Percy
Percy•13mo ago
Project ID: 81d08146-e275-403d-8a25-046edbf806f6
CoCa
CoCa•13mo ago
81d08146-e275-403d-8a25-046edbf806f6 Do I not need to provide a scheme (http or https) when calling my backend API? I'm going to try that now
Solution
Brody
Brody•13mo ago
you need to use http, and specify the port
Brody
Brody•13mo ago
Also, the Golang REST API connects to a PostgreSQL (TimescaleDB) instance, and I would like some help figuring out how to connect from my REST API to the database over private network as well!
going to writing a little guide on this
CoCa
CoCa•13mo ago
so http://example.railway.internal:80/api/path ?
Brody
Brody•13mo ago
what port does your go app listen on?
CoCa
CoCa•13mo ago
well crap lol, it looks like it's probably using 8000 as a default
Brody
Brody•13mo ago
then use 8000 in that url this is the private network, there is no proxy and no firewalls, anything a service listens on is exposed within the private network tcp, udp, http, anything, you just need to use the correct scheme, and port when connecting
CoCa
CoCa•13mo ago
okay so it turns out that railway was automatically setting a port for my backend dynamically and I've updated my api to listen on 80. GitHub is having some issues right now so my changes aren't rolling out but that should fix the UI -> API communication over the internal network, next is the database which I'll handle once there's a guide for it 🙂
Brody
Brody•13mo ago
haha you could have just set a fixed PORT variable
CoCa
CoCa•13mo ago
yep exactly, and that's what I just ended up doing 😅
Brody
Brody•13mo ago
you should use a non privileged port like 8080 still working on that guide though
CoCa
CoCa•13mo ago
I just noticed the PG over Private Network guide in a separate thread, and I'm not sure I'll be able to utilize that until I migrate away from using the Postgres Plugin Database and onto the volume-based database
Brody
Brody•13mo ago
you are correct
CoCa
CoCa•13mo ago
something I've been putting off for weeks but it looks like I'll have to make some time for this very soon
Brody
Brody•13mo ago
railway is soon coming out with a one click migrate button
CoCa
CoCa•13mo ago
😮 oh man that's awesome, I can probably wait until that's available cause plugins are sunsetting soon right?
Brody
Brody•13mo ago
or you can do it now semi-manually with https://railway.app/template/postgres-plugin-migration
Brody
Brody•13mo ago
did the guide help? 🙂
CoCa
CoCa•13mo ago
I'll get back to you once I've used it lol - I'm currently debating on if when I want to move ahead with the plugin migration 😅 I know I really should get this taken care of as soon as possible, I just have other things I'm focusing on at the moment But the private network between my UI & REST API already basically halved my egress, so I'm happy for now
Brody
Brody•13mo ago
awsome, let me know if you run into any issues during the migration
CoCa
CoCa•13mo ago
I did run into an issue using the migration template:
ERROR: relation "_timescaledb_catalog.metadata" does not exist
ERROR: relation "_timescaledb_catalog.metadata" does not exist
From this here in the script - https://github.com/railwayapp-templates/postgres-plugin-migration/blob/main/migrate.sh#L114-L115 The new Postgres DB isn't a timescale image, so it should probably skip that line?
Brody
Brody•13mo ago
yeah ill be letting jr know this
CoCa
CoCa•13mo ago
i'll open a PR that adds a condition check for the table, but maybe it's best to just remove those lines? Is TimescaleDB even an option in the new volume-based databases?
Brody
Brody•13mo ago
railway provides a TimescaleDB template, its just not the default https://railway.app/template/VSbF5V
CoCa
CoCa•12mo ago
there were also some of these warning showing up on the dump
pg_dump: warning: there are circular foreign-key constraints on this table:
pg_dump: detail: hypertable
pg_dump: hint: You might not be able to restore the dump without using --disable-triggers or temporarily dropping the constraints.
pg_dump: hint: Consider using a full dump instead of a --data-only dump to avoid this problem.
pg_dump: warning: there are circular foreign-key constraints on this table:
pg_dump: detail: chunk
pg_dump: hint: You might not be able to restore the dump without using --disable-triggers or temporarily dropping the constraints.
pg_dump: hint: Consider using a full dump instead of a --data-only dump to avoid this problem.
pg_dump: warning: there are circular foreign-key constraints on this table:
pg_dump: detail: hypertable
pg_dump: hint: You might not be able to restore the dump without using --disable-triggers or temporarily dropping the constraints.
pg_dump: hint: Consider using a full dump instead of a --data-only dump to avoid this problem.
pg_dump: warning: there are circular foreign-key constraints on this table:
pg_dump: detail: chunk
pg_dump: hint: You might not be able to restore the dump without using --disable-triggers or temporarily dropping the constraints.
pg_dump: hint: Consider using a full dump instead of a --data-only dump to avoid this problem.
i'll skip doing a PR, I don't really have a great way to test out any changes 😅 it can wait I'm currently running the Postgres Plugin Migration to migrate my plugin DB to a new volume-based Postgres DB - The migration is running ( I think ) but the plugin migration service is regularly capped at 32GB Memory - I suspect this is the sed -i commands commenting out the timescaledb plugin creation? I haven't seen any data come into the new DB yet, so maybe I'll just let this run for a bit? I'm just worried about it being steadily at 32GB Memory
Brody
Brody•12mo ago
would you say you have a lot of data in your database?
CoCa
CoCa•12mo ago
the database itself is ~11GB - but the log line containing the dump says its 30G, so that's what leads me to suspect the sed -i commands?
Brody
Brody•12mo ago
yeah guess sed is editing the file in memory? @jr - enormous memory usage from migration service
jr
jr•12mo ago
hmm yes that is a problem. Taking a look Feel free to delete the migration service if it is taking way too long. I will report back once that has been patched
CoCa
CoCa•12mo ago
I'm going let it keep going just in case there's any other issues
Brody
Brody•12mo ago
youre charged minutely usage for memory, id recommend closing it as it has already likely locked up
CoCa
CoCa•12mo ago
okay, good call - i'll stop it
CoCa
CoCa•12mo ago
here's what I was seeing, it looks like it took some time to finish but these are what I believe are the sed -i spikes?
No description
Brody
Brody•12mo ago
i wouldnt trust the data on the new database if i where you
CoCa
CoCa•12mo ago
these are the metrics of the plugin migration service, i canceled it and my target DB is still empty and by "took some time to finish" - i mean the sed commands
jr
jr•12mo ago
It doesn't look like there was any data transfered. It was the sed commands. I'm working on replacing those with something more efficient to handle massive files I've replaced sed with awk in the migration script which should be more memory efficient. I'm going to add some more optimizations, but it shouldn't jump to 32gb of memory. It may still take a bit to process the large database though
CoCa
CoCa•12mo ago
Okay, giving it a fresh run now! 😃
jr
jr•12mo ago
Looks like it was successful!
jr
jr•12mo ago
You may want to restart the new database to clear anything in memory after the restore
No description
CoCa
CoCa•12mo ago
yep, i'll give it a restart, just checking the data is all good locally real quick 🙂
jr
jr•12mo ago
oh yeah for sure. Definitely check everything is all good and share if anything is off
CoCa
CoCa•12mo ago
looks good! now i just gotta take care of my application cutover
Brody
Brody•12mo ago
now you can use the private url!
jr
jr•12mo ago
Keep in mind any data written to the plugin after the migration will not be in the new database. You are also free to keep the legacy plugin around for a bit after cutting over so that you can be very confident everything is there
CoCa
CoCa•12mo ago
after changing the env variables with the private networking info my Go server is having trouble connecting to the database
exiting with error: error opening db: failed to connect to `host=postgres.railway.internal user=postgres database=railway`: hostname resolving error (lookup postgres.railway.internal on 8.8.8.8:53: no such host)
exiting with error: error opening db: failed to connect to `host=postgres.railway.internal user=postgres database=railway`: hostname resolving error (lookup postgres.railway.internal on 8.8.8.8:53: no such host)
I'm using the gorm driver, which takes a DSN formatted string
jr
jr•12mo ago
I think this may be the network initialization time. Can you try adding a small delay (500ms-1s) before the server starts? https://docs.railway.app/reference/private-networking#caveats If it is still happening, it is something else. But we can debug it
CoCa
CoCa•12mo ago
i upped it to a 10s sleep and that seemed to give it enough time still encountered the error on a 1s sleep
jr
jr•12mo ago
ah okay. 10s is a lot and 2s should be okay. You can also add a healthcheck to enusre that your application is up and ready before the old deployment goes down
Brody
Brody•12mo ago
you should only need a max of 3 seconds, the highest time ive ever seen it take was 3.67s
jr
jr•12mo ago
This is also something we are actively working on. Literally a single engineer is dedicated to improving this part of Railway
Brody
Brody•12mo ago
databases as services are cool, but having no initialization time is cooler
CoCa
CoCa•12mo ago
that's good stuff 🙂 I'll definitely need to start making use of healthchecks going forward I think I'm okay with leaving the startup sleep at 10s for now - as far as I know nobody really visits this website haha - it really just exists for me at this point to experiment with stuff
Brody
Brody•12mo ago
the delay would only effect the initial app start time
CoCa
CoCa•12mo ago
yep, and the only time it really starts up is if I'm pushing changes - I've never really had it crash or restart on it's own, it runs pretty smoothly
Brody
Brody•12mo ago
for sure still use a healthcheck, even just a simple one that returns a 200 status code
Happy^!
Happy^!•12mo ago
Hey everyone. Is there a way to accept UDP traffic from public network? and is it possible to expose more than one port?
Brody
Brody•12mo ago
unfortunately 1. nope 2. nope
Want results from more Discord servers?
Add your server