R
Railway•8mo ago
Prodigga

Redis Networking/Egress despite using internal networking

Hi there We are seeing Network activity on our Redis metrics despite connecting to it via internal networking. Is this expected? (Ignore the dip! We were down for maintenance 😆 ) Project ID 4c3b4b0e-006a-407e-90c7-9c3031cd622f
No description
Solution:
Ok - I figured it out. While waiting for DNS changes to propagate I left our Heroku Dyno live. I had it connected to our Railway Redis service via public networking. I figured if I disable public networking it would kill any active connections but it seems like thats not the case? The Heroku Dyno was still connected to the Redis Service. Anyway, shutting down the dyno reduced network usage as expected...
No description
Jump to solution
52 Replies
Percy
Percy•8mo ago
Project ID: 4c3b4b0e-006a-407e-90c7-9c3031cd622f
Brody
Brody•8mo ago
well one way to make sure nothing is connecting to it publicly would be to remove its tcp proxy, then before long something will crash
Prodigga
Prodigga•8mo ago
😆 true! i can do this for our stg and dev servers lets see how it goes oh wait, i think i know why i am using the internal ip but using the public port ie **.rlwy.net:56258 -> :6379 Private Networking redis.railway.internal and I am connecting to it via redis.railway.internal:6379 lol
Brody
Brody•8mo ago
the public port is 56258 in this case, 6379 being the private port
Prodigga
Prodigga•8mo ago
oh, right right then thats not it
Brody
Brody•8mo ago
what you actually described redis.railway.internal:56258 would also just outright not work
Prodigga
Prodigga•8mo ago
yeah i didn't think it would, i was kinda suprised it was working at all with what i described! but yeah seems like I have that the right way around disabling the public TCP proxy now for dev/stg and observing/testing seems fine so far... strange.. there is so much traffic on prd
Brody
Brody•8mo ago
prod is somehow connecting to it via the public network?
Prodigga
Prodigga•8mo ago
hm... i dont think so but we will see i disbled it for prd too and just observing again but both stg and dev still show some traffic, minor, but still will see what happens with prd
Brody
Brody•8mo ago
sounds good
Prodigga
Prodigga•8mo ago
Hmm, nope its like the Redis traffic counts as egress..?
Prodigga
Prodigga•8mo ago
been about 10 mins
No description
Brody
Brody•8mo ago
the traffic on the private network shouldn't be measured, check the cost breakdown to see if your redis service is incurring any egress
Prodigga
Prodigga•8mo ago
good idea
Prodigga
Prodigga•8mo ago
No description
Prodigga
Prodigga•8mo ago
yeah it is :blobconfused: and 'double the cost' i guess since the service talking to it is also taking the egress hit
Brody
Brody•8mo ago
and you have completely removed its tcp proxy?
Prodigga
Prodigga•8mo ago
yep!
Prodigga
Prodigga•8mo ago
No description
Brody
Brody•8mo ago
is redis somehow sending data elsewhere? there's no tcp proxy but that doesn't disconnect it from the internet
Prodigga
Prodigga•8mo ago
I am not sure how i would check that. I just spun up redis using right click > database > redis, nothing bespoke
Brody
Brody•8mo ago
what are you using redis for?
Prodigga
Prodigga•8mo ago
various things - caching, temp store, used by my redis io adapter each has their own, and they are the only ones communicating with their own redis db lol
Prodigga
Prodigga•8mo ago
No description
Brody
Brody•8mo ago
okay well now I have to go off on a tangent here why service groups for your different environments instead of project environments?
Prodigga
Prodigga•8mo ago
hahaha i was debating whether i should put this up because i figured you would mention that haha is there a cross-project egress cost? because my API Gateway would have to be in its own project then the game servers in a nother, with an environment for each but it means all comms will happen from API Gateway <- across projects -> Game Server
Brody
Brody•8mo ago
well you would have a single project, with multiple environments, each environment would have a gateway, game server and it's redis database
Prodigga
Prodigga•8mo ago
yeah but the gateway directs traffic to one of the 3 environments so which is the 'true true' gateway, and are the other 2 just dummies that dont do anything, i guess? haha
Brody
Brody•8mo ago
if you where to go with the architecture I mentioned, your gateway would only route traffic to a production (or set of production) game server services in the production environment, and the gateway in the development environment would route traffic to the development game server in the same environment
Prodigga
Prodigga•8mo ago
it makes most sense to me if i could nest this setup inside one another.. something like
Main Project - contains Gateway | -> Dev/Stg/Prd
Game Server Project - contains game server/redis database. | -> Dev/Stg/Prd |
Main Project - contains Gateway | -> Dev/Stg/Prd
Game Server Project - contains game server/redis database. | -> Dev/Stg/Prd |
yeah but I need the single gateway to be able to route traffic to the Dev or Stg or Prd version of the server
Brody
Brody•8mo ago
no matter how you do it, you absolutely wouldn't want to be using multiple projects, and honestly with the group UI you have done it quite cleanly, was just curious is all basically I'm stalling while trying to think of potential reasons for seeing such high outbound usage
Prodigga
Prodigga•8mo ago
it would be cool to be able to nest a project inside another, and then from the parent project I could have an API gateway to proxy the traffic to different environments of the sub project (dev/stg/prd) and then I could spin up a dev version fo this WHOLE setup and test the entire stack because right now I don't really have a 'stg' or 'dev' version of the my API gateway
Brody
Brody•8mo ago
Prodigga
Prodigga•8mo ago
For that, I could use the environments feature. Have a Dev/Stg version of my current project that, inside it, has the Dev/Stg/Prd game services 'baked in' hah! right? its a lot of traffic we do run a multiplayer game so there is a lot of traffic going in and out, so I think the volume maybe makes sense
Brody
Brody•8mo ago
does the game server even have comparable traffic?
Prodigga
Prodigga•8mo ago
yeah, much more i am grappling with this now, as our egress costs have made hosting on railway much more expensive than heroku hah
Brody
Brody•8mo ago
the gateway doesn't touch redis at all right? yeah unfortunately there's not much railway can do about that right now since GCP has very high egress costs
Prodigga
Prodigga•8mo ago
so i am trying to make sure i am doing everything right and only incuring egress from what is absolutely neccessary, to get a better cost estimate. and thats how i encountered this odity with Redis +1 yep
Brody
Brody•8mo ago
railway is moving to bare metal, I suspect the egress costs will drop and they will hopefully pass that down to the users too but I am genuinely stumped here
Prodigga
Prodigga•8mo ago
fingers crossed, yeah it is a bit too pricing, it stings. I was excited about railway for future projects - for hosting game servers and such, but it might just cost too much in terms of egress for realtime applications
Prodigga
Prodigga•8mo ago
here is our prd game server
No description
Brody
Brody•8mo ago
it might sting less if redis didn't have traffic for some reason :mildpanic:
Prodigga
Prodigga•8mo ago
exactly! hah 🙂
Brody
Brody•8mo ago
but good news, you're pro so I can escalate this to the team, hopefully they can dig into some of the internal network logs for these services, any last ideas before i do that though?
Prodigga
Prodigga•8mo ago
i just double checked my redis urls and such but i guess with public networking disabled it doesnt mean much so no, not really. I am also stumped! thanks 🙂
Brody
Brody•8mo ago
the team when available will communicate via that help station thread I've heard they love when I escalate long debugging threads 🙂
Prodigga
Prodigga•8mo ago
😄 awesome thanks so much, curious to see what the outcome will be
Brody
Brody•8mo ago
me too!
Solution
Prodigga
Prodigga•8mo ago
Ok - I figured it out. While waiting for DNS changes to propagate I left our Heroku Dyno live. I had it connected to our Railway Redis service via public networking. I figured if I disable public networking it would kill any active connections but it seems like thats not the case? The Heroku Dyno was still connected to the Redis Service. Anyway, shutting down the dyno reduced network usage as expected
No description
Brody
Brody•8mo ago
how was the dyno generating so much network usage lol
Prodigga
Prodigga•8mo ago
Hmm, yeah, I am not sure why there is so much inbound traffic. I sort of expect the outbound. We're using the Redis Adapator with Socket IO to stream realtime multiplayer gameplay from Railways to the Heroku Dyno anyway, I'll have to make sure I didn't write something too stupid on my end. but atleast the egress issue is solved tldr user error, though it is curious that the dynos connection to the Redis DB was not 'severed' when I disabled public networking
Brody
Brody•8mo ago
i will ask the person who wrote the tcp proxy if that would even be possible but glad you where able to solve! all the more reason to shut down all your dynos 😉
Want results from more Discord servers?
Add your server