Shared private IP/domain between all services?
Is there a shared private IP that all the containers in a project use? I have a use case where I need to set an IP/address that can be used to broadcast information to other containers/services. But my other services have all different domains (*.railway.internal), which won't work.
118 Replies
Project ID:
196487ef-2f0e-4749-9286-a107d2c9a8a8196487ef-2f0e-4749-9286-a107d2c9a8a8
all ips are dynamic, both public and private, why wont the internal domains work for you?
Just a requirement of Temporal I think? I'm working on splitting the Temporal container (that included the frontend, history, matching, and worker services) all up into their own containers. There's a
TEMPORAL_BROADCAST_ADDRESS that you have to set. This is how the services find one another I guess.. to share membership info? I tried setting it to 127.0.0.1 but obviously that won't work ๐try setting it to
[::] or :: https://docs.railway.app/guides/private-networking#communicating-over-the-private-networkOh lemme try ๐
Hmm nope ๐ฆ
Here's my updated template https://railway.app/new/template/CnHq4I and repo https://github.com/dihmeetree/railway-temporal
btw things like this go in the service variables (or template variables) https://github.com/dihmeetree/railway-temporal/blob/main/nginx/Dockerfile#L5
Yeah I know.. I haven't focused on the variables yet
Just trying to get it working first ๐
i mean it could matter
oh really?
well its in the wrong place, it does nothing there
I have it set as a variable already tho.. in the template
I think it's just there as a duplicate ๐
its not on the nginx service at least
I'm not using nginx at the moment.. trying to get it working directly first
i know, but maybe nginx isnt the only service that youve not added this variable to?
I just checked when you said it
It's there on the other things ๐
it is indeed
do all the hosts have the same internal domain?
How would I find that out?
Also one other thing to note
In my Nginx container I get the error
I think that's the issue I was getting yesterday when I decided to stop using it
yeah theres a trick to getting nginx to work with internal domains
๐ฎ
There's always tricks ๐
um haha i dont know, youre the one who would know more about this than me
Highly debatable ๐
yeah its called just use caddy (caddy works out of the box but nginx can work too)
Oh yeah I tried using Caddy but
I couldn't find good documentation about how to do grpc with it
Well.. Caddy isn't even my biggest issue rn tho..
I can't even get stuff to work directly
If I can get it working without Caddy then I can look into that after
so, do you know how to figure this out?
I think this is mostly a Temporal blocker though. Because yes.. there is a way to connect to all other nodes (if I had the ability to specific like each node via their internal domain right? *.railway.internal) But this
TEMPORAL_BROADCAST_ADDRESS specific field is kinda forcing me to specify either an IP or (maybe domain?) that allows me to share info with all services that share that IP/domaindoes a single service start a bunch of workers? are you using replicas?
No each service of Temporal is in it's own service on Railway
I haven't done anything with replicas/scaling yet
theres 4 host:ports here, is a single service running 4 workers?
No that's logs from 1 of the services. I believe those are the 4 "IP's" the service is trying to reach out to.
Each has it's own port
So for example (frontend:port, history:port, matching:port, and worker:port)
I think Fly.io has something where you can just alias your entire app.. and it let's you communicate to all your machines? I think?
That's why I was asking if Railway had something like that
can the temporal server only run one service type at a time?
i mean the env is plural
SERVICESI believe you could run multiple yes
then do that, that way you will be able to use a single internal domain
That's what I had before ๐ It worked fine yes, but i'm trying to "productionify" by splitting the services into their own containers
but you do might need to bind on
:: still or it might never work no matter what you do
https://github.com/dihmeetree/railway-temporal/blob/main/temporal/matching/Dockerfile#L7
https://github.com/dihmeetree/railway-temporal/blob/main/temporal/matching/Dockerfile#L7So each individual service can be scaled differently if needed
Oh ๐ฎ
Would I broadcast on 127.0.0.1 then? If Bind is
:: ?i get where your coming from, but if temporal doesnโt allow you to specify separate hosts for individual services, i dont think you have many options besides proxy, and thats a meh solution
Yeah no thanks to proxy ๐
uh kinda, maybe check out a article on the differences of 127.0.0.1, 0.0.0.0, ::1, ::, etc
Well yeah if it doesn't work on Railway it doesn't work.. no big deal
Was just trying to get it working
i really wouldnโt call this a limitation of railway, id say this was a suboptimal design by not allowing you to specify individual addresses for each service type
Here's what the Temporal guy said
thats a compose file isnt it?
Yeah
well thats not too helpfull lol
cc @Melissa im 99% sure this is the template they asked to move over?
haven't read through everything in here yet, but we do have this other temporal template (https://github.com/railwayapp-templates/temporal), perhaps it could be helpful to reference?
they originally did fork that template, all due respect that template left a lot to be desired
the variables are also setup sub optimally from what i remember
yeah, lack of reference variables, heres just two examples
ahh got it got it
dmitry has fixed that stuff in their template now though
Yeah the original Temporal template was missing ElasticSearch as well, which plays a key role in serving search queries and such on the Temporal UI. I was able to add that and had everything working. The issue at the moment comes down to splitting the Temporal services (from the 1 Temporal service with the frontend, history, matching, and worker) into their own Railway services.
Breaking the Temporal services into their own Railway services would let you scale individual Temporal services when it came time to using it in production. For example you may need to scale the Temporal "history" service more than the "frontend" service for example, or etc etc.
Which is what I've been trying to figure out.. but hit a snag ๐
My template is kinda "under construction" you can say... it doesn't utilize proper Railway variables and such yet in terms of the template. Still trying to get it working functionally.. then I can go back and make everything look pretty ๐
really interested in a scalable Temporal template, if there's any demo I can try would be nice :)
Yeah i'm working on it ๐
Will let you know ๐๐ป
really cool, let's get this working then! I am in between a couple of things right now, but I can try to hack on it with y'all in a bit
That would be sweet! Appreciate it ๐๐ป
what service in this template is trying to connect to all these frontend, matching, worker, etc services?
Also appreciate your help as well Brody! ๐
So each service .. Don't quote me on this.. I believe broadcasts information to the other services. I believe it's over a single specific IP; the
TEMPORAL_BROADCAST_ADDRESS variable I was referring to before. And each of the services have a specific different port that they use to communicate to each other with?
I'm new to Temporal interworkings to be honest.. it's not like i've been using this forever. I'm trying to learn how it works too ๐Here's a Docker compose to get Temporal working locally https://github.com/temporalio/docker-compose/blob/main/docker-compose-multirole.yaml
GitHub
docker-compose/docker-compose-multirole.yaml at main ยท temporalio/d...
Temporal docker-compose files. Contribute to temporalio/docker-compose development by creating an account on GitHub.
This docker compose template works great locally ๐
It's what i've been using to split stuff up
They have 2 frontend services tho.. not sure why. But I believe only 1 is needed.
https://docs.temporal.io/assets/images/temporal-cluster-30b133bd4034cd3226bf908ed3810e45.svg (From their Docs)
The Temporal Server consists of four independently scalable services:
Frontend gateway: for rate limiting, routing, authorizing.
History subsystem: maintains data (mutable state, queues, and timers).
Matching subsystem: hosts Task Queues for dispatching.
Worker Service: for internal background Workflows.
For live (production) environments, we recommend that each service runs independently, because each one has different scaling requirements and troubleshooting becomes easier. The History, Matching, and Worker Services can scale horizontally within a Cluster. The Frontend Service scales differently than the others because it has no sharding or partitioning; it is just stateless.
Each service is aware of the others, including scaled instances, through a membership protocol via Ringpop.
So I removed
BIND_ON_IP and TEMPORAL_BROADCAST_ADDRESS and discovery does seem to be working? I'm assuming these IP's are of the containers?
But i'm still getting an error from the History service.. which is trying to connect to the Frontend service over port 7233
6933 is the Frontend Service, 6934 is the History Service, 6935 is the Matching Service, 6939 is the Worker Servicethose are ipv4 addresses from the eth0 interface, don't think thats going to work
oh 
the private network is ipv6 only on railnet0
How do I broadcast like the unique ipv6 address from each container then? ๐ค
I have to like somehow broadcast that
as for the connection refused while connecting to the frontend, make sure you are connecting to the correct port and that your frontend is binding on
[::]I tried setting
ENV BIND_ON_IP="[::]" but I was getting an errorthat's just binding, you need to specify the correct address and port to connect to elsewhere
The
history service is most likely connecting to this ENV TEMPORAL_ADDRESS="temporal-frontend-service.railway.internal:7233"
Is that not correct? ๐คyou tell me, what is the internal address of the frontend? and what port is it listening on
So there's 4 services. Each one I tried setting
ENV BIND_ON_IP="[::]" Each start on their own port. But the frontend port is 7233have you been able to confirm the frontend is indeed listening on 7233
Yes in the logs it prints
Use rpc address 127.0.0.1:7233 for cluster active.
err.. that's with the bind on [::] removed ๐well you definitely don't want it to start on 127.0.0.1
You wouldn't be up for a Discord voice call would you by chance? ๐
I wouldn't, sorry
Ok no worries ๐
So yeah
ENV BIND_ON_IP="[::]" doesn't work. Temporal throws an error:
try without the square brackets
I do wanna mention this now for when you get this working, all these ENVs in the dockerfile should be transferred to service variables
though you probably already know that
Yep! Like I said.. just trying to get it working first ๐
It doesn't like
ENV BIND_ON_IP="::" either:
but you're doing stuff with environment variables right now, why is it mentioning a yaml file?
I don't know ๐ญ
GitHub
Improve ipv6 support (#4766) ยท temporalio/temporal@e62c22f
Temporal works with ipv6, but there are some clarifications in documentation and additional settings that could make support cleaner. Specifically:
* specify pprof host to override localhost def...
This will be included in the next release ๐
GitHub
[temporalio/auto-setup] bindOnIP does not work with ipv6 ยท Issue #3...
Reopening #3612 Expected Behavior I can bind to ipv6 address. Actual Behavior '[2a05:d014:6e0:c883:1d45::11]:7233' you can see in the log. This address is using correctly for listening but ...
oof
wut
youre on 1.22.2
though that issue is from spet/oct, next release could have already happened a few releases ago depending on how often they do releases
hmm lemme try with the latest release
it's not on dockerhub yet ๐
Hmm actually idk if they even merged that PR into a release? ๐ค
I don't see it in a release
#4766
i dont either
Apparently the changes will be in 1.23.0
thats good haha
Brody, when I run this command in my Docker buildstep.. it returns blank..
Is there a way I can get the specific Railway instance's private IP?
the private network is not available at build
oh ๐
oh ye that makes sense duh
crap nvm come it's not exposed as a official environment variable ๐ค
come it's not exposed as a official environment variable ๐คwhat? the ipv6 address?
yeah
because you should use the domain lol
I don't think BIND_ON_IP will accept the domain ๐ค
but I can try
"::" wasn't working (not sure why) so I was gonna try to see if I could get the direct private ipv6 IP of the instance instead
Having SSH access to the instances would make my life so much easier
:: is the correct way, the app not accepting that is not fault of railways
at the surface a lot of people think having ssh access to the container will instantly solve a lot of problems, and it could, and there are valid usecases for it, but I'm my experience it would bring far more issues (thus help threads) from misuse or improper use, so I strongly believe that the containers should stay without ssh access.Yeah you're totally right ๐
I've got 3 IP's here.. which one would I bind to, if I want them to all talk to eachother?
The
railnet ipv6 address it the one that all instances can talk on or? ๐คrailnet0 is the interface for the wireguard tunnel, but again you want to bind these apps to
:: and not their own ip addresses, i strongly believe you are chasing the wrong goose here.
if they dont support binding to :: or [::] then it just wont work, even if you worked out a way to give them their own ip addresses to bind to.Yeah so I found out how they build their docker images https://github.com/temporalio/docker-builds
GitHub
GitHub - temporalio/docker-builds: Temporal service Docker images b...
Temporal service Docker images build. Contribute to temporalio/docker-builds development by creating an account on GitHub.
i'm able to build the 2 images I need
and fix any issues in the code potentially
just wait for them to release the version with ipv6 support lol
Well I have the code from the main branch (with the ipv6 "fix")
It's still not working with "::"
do you get a different error?
No same error
It doesn't like
:: or even [::] for that matterthen its also possible
0.0.0.0 will bind to ipv6 toolong shot, but maybe yaml is treating
:: as a property declaration? is it surrounded by quotes?i have had this on my list to come back to forever!! were y'all able to get this working, or blocked by ipv6?