RTT to nearby datacenter has gone way up in the past few days

Regions have been amazing and are super helpful for our use case. We host our database in AWS us-east-1, which is in the Washington DC area. We've moved our Railway servers to GCP us-east-4 as that is in the same local area. I was experimenting with this last week and was getting single digit to low double digit millisecond RTT/latency from Railway to AWS. This weekend I redeployed the same service and found that the latency has gone back up to >70 ms round-trip time. I'm scratching my head here, because when I can double check the IP of my Railway service and AWS database and see they are both in the Washington DC area, yet this latency is really bad for that proximity. My only thought is perhaps in your development work you all have added some configuration that adds some sort of an additional layer to this networking that adds additional latency. What could be going on here?
11 Replies
Percy
Percy14mo ago
Project ID: 95c304a2-fe2b-4ad1-bc3f-5296fd26f36c
Ian Woodfill
Ian Woodfill14mo ago
95c304a2-fe2b-4ad1-bc3f-5296fd26f36c
Ian Woodfill
Ian Woodfill14mo ago
Medium
Public Cloud Inter-region Network Latency as Heat-maps
Optimize your public cloud region choices.
Ian Woodfill
Ian Woodfill14mo ago
For context, my local instance has RTT of ~20ms, and I'm in NY. Theres definitely some funky networking going on here, given gcp-us-east4 (railway) <-> aws-us-east-1 is over 3x slower than NY <-> aws-is-east-1, and given that last week it was very low latency actually, resolving the IP address for my services shows that they're still in us-west, despite having RAILWAY_REGION=us-east4 oh, it seems like RAILWAY_REGION does not work utilizing a shared variable! This is resolved
Brody
Brody14mo ago
for future reference, the region in use is printed in the first line in the deployment logs
Ian Woodfill
Ian Woodfill14mo ago
yes it is! good to know
No description
Ian Woodfill
Ian Woodfill14mo ago
do you know how volumes work with regions? what happens if you change the region of a service that has a volume?
Brody
Brody14mo ago
https://docs.railway.app/deploy/deployments#caveats while your question is a little different than the wording used in the docs, I think they are similar enough to not risk it without first testing it in a test project
Ian Woodfill
Ian Woodfill14mo ago
ah
Brody
Brody14mo ago
I'd also be interested to know the results of that test!
Ray
Ray14mo ago
We’re discussing this next week 🙂 Right now they don’t work. If you set a region on a svc with volume attached, it’ll be stuck at the deploy phase Primarily because your volume can be on a different box than what it was on when switching regions
Want results from more Discord servers?
Add your server