R
Railway•14mo ago
drdrigg

EU servers are slower than US server (I am in the EU)

I have deployed an empty Postgres template, a Redis template and a Rails app in the European datacenter and it responds in 350 ms. But when I have the servers in the US West the same endpoint responds in 260 ms. I am located in Europe. How can the response be slower when using EU server (I am in the EU) than using US servers? Project ID: 988851b6-78d5-44de-a73d-3e5b6468c8bf
78 Replies
Percy
Percy•14mo ago
Project ID: 988851b6-78d5-44de-a73d-3e5b6468c8bf
Brody
Brody•14mo ago
@char8 I think you'd be the person to answer this 🙂
luna
luna•14mo ago
There’s a good chance this is fast pathing causing issues https://discord.com/channels/713503345364697088/1067670962276945961/1156374680937177098
Brody
Brody•14mo ago
I agree, but wouldn't hurt to hear char's opinion
luna
luna•14mo ago
Just figured I’d mention it here incase it was missed.
drdrigg
drdrigg•14mo ago
No this cannot be it. I have tried three independant ISPs (2 wifi and one celluar), same problem. Also I used AWS (which you use) many times before on both continents and never experienced this.
Brody
Brody•14mo ago
railway uses exclusively gcp at this time
drdrigg
drdrigg•14mo ago
Google Cloud Platform? The strange thing is that when using EU servers the response time should not be over 300 ms. I use the Vultr clouds for now (investigating if I can move this app to Railway) and the response time there is 111 ms for the same endpoint. Railway servers is 420 ms. Could you check so that I have configured it correctly? Also the railway database is almost empty (1 record) and the Vultr one is over 2 GB.
Brody
Brody•14mo ago
what's the hostname of your database
drdrigg
drdrigg•14mo ago
Is it this? roundhouse.proxy.rlwy.net:11179
Brody
Brody•14mo ago
yep that's definitely using the new database templates that supports regions
drdrigg
drdrigg•14mo ago
Yes I really want to use Railway for this project but four times slower is not ok unfortunately
Brody
Brody•14mo ago
and you've set the region with the drop-down and confirmed it deployed to the correct location by looking at the first log line of the build?
drdrigg
drdrigg•14mo ago
Yes I used the dropdown. Only the Rails app Build log contains anything, not the Postgres and Redis (but all dropdowns says EU servers).
No description
No description
No description
char8
char8•14mo ago
if you make a bunch of requests, say 5-15 ish, do they all return at 350ms or or do you see a spread? I'm wondering if you're hitting the cache warming on the tcp proxies (the first couple of requests to each database will incur some high tail latencies) Also can you list the callpath that's showing the 350ms? Eg: is it between you and your app or is it multi hop, eg: you <-> app <-> redis/postgres I checked and everything is correctly deployed to the EU
drdrigg
drdrigg•14mo ago
Here is a few requests after each other on the Railway servers.
No description
No description
No description
No description
drdrigg
drdrigg•14mo ago
These are requests to the same codebase and endpoint and user but to my Vultr servers.
No description
No description
No description
No description
drdrigg
drdrigg•14mo ago
They are also located in Amsterdam Also the Vultr servers are pretty heavy used and my Railway server only used by me.
char8
char8•14mo ago
yeah this is odd - I'm going to write up a quick benchmark app to double check and get back. You could try moving service <-> db communication over to private networks, those will bypass our TCP proxies and direct dial the db (so just 1 network hop).
drdrigg
drdrigg•14mo ago
How can I do that?
Brody
Brody•14mo ago
reference the DATABASE_PRIVATE_URL instead
drdrigg
drdrigg•14mo ago
In service variables? So remove database_url and add DATABASE_PRIVATE_URL?
Brody
Brody•14mo ago
something like that yeah
drdrigg
drdrigg•14mo ago
There is no such option
No description
drdrigg
drdrigg•14mo ago
Where can I find the value for DATABASE_PRIVATE_URL?
Brody
Brody•14mo ago
hmmm you may have deployed the template before the private variable was added one sec DATABASE_URL=${{Postgres.RAILWAY_PRIVATE_DOMAIN}} use that on the service that connects to the database
drdrigg
drdrigg•14mo ago
Ok redeploying now ${{Postgres.RAILWAY_PRIVATE_DOMAIN}} Saves as: <Empty string>
Brody
Brody•14mo ago
can I see a screenshot of your project?
drdrigg
drdrigg•14mo ago
No description
Brody
Brody•14mo ago
okay you did it a little differently than what I had in mind DATABASE_URL=${{RAILWAY_PRIVATE_DOMAIN}} try that instead
drdrigg
drdrigg•14mo ago
Added like this (raw) DATABASE_URL=${{RAILWAY_PRIVATE_DOMAIN}} PGDATA=${{Postgres template.PGDATA}} PGDATABASE=${{Postgres template.PGDATABASE}} PGHOST=${{Postgres template.PGHOST}} PGPASSWORD=${{Postgres template.PGPASSWORD}} PGPORT=${{Postgres template.PGPORT}} PGUSER=${{Postgres template.PGUSER}} POSTGRES_DB=${{Postgres template.POSTGRES_DB}} RAILS_LOG_TO_STDOUT=true REDISHOST=${{Redis template.REDISHOST}} REDISPASSWORD=${{Redis template.REDISPASSWORD}} REDISPORT=${{Redis template.REDISPORT}} REDISUSER=${{Redis template.REDISUSER}} REDIS_PASSWORD=${{Redis template.REDIS_PASSWORD}} REDIS_URL=${{Redis template.REDIS_URL}}
Brody
Brody•14mo ago
okay now you went back to the original way I wanted you to do it so that's now wrong in the service that Accesses the database DATABASE_URL=${{Postgres.RAILWAY_PRIVATE_DOMAIN}}
drdrigg
drdrigg•14mo ago
Changed, deploying Got this now, the database_url is empty: DATABASE_URL=${{Postgres.RAILWAY_PRIVATE_DOMAIN}} PGDATA=${{Postgres template.PGDATA}} PGDATABASE=${{Postgres template.PGDATABASE}} PGHOST=${{Postgres template.PGHOST}} PGPASSWORD=${{Postgres template.PGPASSWORD}} PGPORT=${{Postgres template.PGPORT}} PGUSER=${{Postgres template.PGUSER}} POSTGRES_DB=${{Postgres template.POSTGRES_DB}} REDISHOST=${{Redis template.REDISHOST}} REDISPASSWORD=${{Redis template.REDISPASSWORD}} REDISPORT=${{Redis template.REDISPORT}} REDISUSER=${{Redis template.REDISUSER}} REDIS_PASSWORD=${{Redis template.REDIS_PASSWORD}} REDIS_URL=${{Redis template.REDIS_URL}} Get a 503 error now
Brody
Brody•14mo ago
oh you've named your postgres database "Postgres template" please update the namespace in the variable I gave you
Adam
Adam•14mo ago
@drdrigg was it you that mentioned you’re testing all this on wifi? Do you have an ethernet connection available to test on? it will give much more consistent results and eliminate many user error variables
drdrigg
drdrigg•14mo ago
What should I rename the template to?
Brody
Brody•14mo ago
the name is fine, I was just giving you a reference that uses the namespace of what the default name would be, you can just update the name in the reference to reflect your service name
drdrigg
drdrigg•14mo ago
What should I rename then?
Brody
Brody•14mo ago
DATABASE_URL=${{Postgres template.RAILWAY_PRIVATE_DOMAIN}} see, you are using the namespace of the name of the postgres service, then referencing it's private networking domain
drdrigg
drdrigg•14mo ago
Testing now
Brody
Brody•14mo ago
I assume the variable is rendering properly now?
drdrigg
drdrigg•14mo ago
No description
drdrigg
drdrigg•14mo ago
Now PGhost and DATABASE_URL is the same Not sure I am using the correct variables 🙂 Its a bit confusing
Brody
Brody•14mo ago
100% agree with you, this definitely can become very confusing, the team knows this and will be looking to improve their documentation
drdrigg
drdrigg•14mo ago
Should I remove all the variables? And if yes, which exactly should I add back?
Brody
Brody•14mo ago
hold on one sec, I screwed up, too early for me lol set/overwrite these variables in your postgres service itself
DATABASE_URL=postgres://${{PGUSER}}:${{POSTGRES_PASSWORD}}@${{RAILWAY_TCP_PROXY_DOMAIN}}:${{RAILWAY_TCP_PROXY_PORT}}/${{PGDATABASE}}
DATABASE_PRIVATE_URL=postgres://${{PGUSER}}:${{POSTGRES_PASSWORD}}@${{RAILWAY_PRIVATE_DOMAIN}}:5432/${{PGDATABASE}}
DATABASE_URL=postgres://${{PGUSER}}:${{POSTGRES_PASSWORD}}@${{RAILWAY_TCP_PROXY_DOMAIN}}:${{RAILWAY_TCP_PROXY_PORT}}/${{PGDATABASE}}
DATABASE_PRIVATE_URL=postgres://${{PGUSER}}:${{POSTGRES_PASSWORD}}@${{RAILWAY_PRIVATE_DOMAIN}}:5432/${{PGDATABASE}}
in the service that accesses the postgres service
DATABASE_URL=${{Postgres template.DATABASE_PRIVATE_URL}}
DATABASE_URL=${{Postgres template.DATABASE_PRIVATE_URL}}
I know how this all works and I still messed it up but those new variables should be correct
drdrigg
drdrigg•14mo ago
Server keeps crashing With new variables
Brody
Brody•14mo ago
do they all render properly? and can you please provide the error they fail with, I have an idea of why the connection fails
drdrigg
drdrigg•14mo ago
Ok will check Well there is values in all when I click the eye icon
Brody
Brody•14mo ago
okay and let's see the full errors please
drdrigg
drdrigg•14mo ago
No description
drdrigg
drdrigg•14mo ago
I tried to bundle exec db:create (to create the database since it said it was not created). Will try without this start command.
Brody
Brody•14mo ago
very not good error message lol
drdrigg
drdrigg•14mo ago
Its the gist of it haha
No description
drdrigg
drdrigg•14mo ago
It cannot create the DB (not sure why it needs to do that since it was there before) So I guess it has no connection to the DB Get 503 now
Brody
Brody•14mo ago
can you add a 3 second delay in before the connection to the database attempt? private networking takes some time to initialise
drdrigg
drdrigg•14mo ago
Where can I add that?
Brody
Brody•14mo ago
in code would probably be easiest maybe even sleep 3 && <your normal start command> in the settings menu
drdrigg
drdrigg•14mo ago
Will try On it Made no difference
Brody
Brody•14mo ago
still error?
drdrigg
drdrigg•14mo ago
Must be an error somewhere with the database connection because my code outputs: {"status":500,"error":"There was a server error","details":"FATAL: database "postgres.railway.internal" does not exist\n"}
Brody
Brody•14mo ago
are you building with a dockerfile?
drdrigg
drdrigg•14mo ago
Must be a variable issue (the system does not know what the db name is). Yes
Brody
Brody•14mo ago
what image do you use
drdrigg
drdrigg•14mo ago
This is the content of my Dockerfile: FROM ruby:2.6.8 Set environment variables ENV RAILS_ROOT /app ENV RAILS_ENV production Set working directory WORKDIR $RAILS_ROOT Add Gemfile to the image COPY Gemfile Gemfile.lock ./ Install gems RUN bundle install --without development test Copy the main application. COPY . . Start the application CMD ["bundle", "exec", "rails", "server", "-b", "0.0.0.0"]
Brody
Brody•14mo ago
please enclose it in triple backticks
drdrigg
drdrigg•14mo ago
FROM ruby:2.6.8

# Set environment variables
ENV RAILS_ROOT /app
ENV RAILS_ENV production

# Set working directory
WORKDIR $RAILS_ROOT

# Add Gemfile to the image
COPY Gemfile Gemfile.lock ./

# Install gems
RUN bundle install --without development test

# Copy the main application.
COPY . .

# Start the application
CMD ["bundle", "exec", "rails", "server", "-b", "0.0.0.0"]
FROM ruby:2.6.8

# Set environment variables
ENV RAILS_ROOT /app
ENV RAILS_ENV production

# Set working directory
WORKDIR $RAILS_ROOT

# Add Gemfile to the image
COPY Gemfile Gemfile.lock ./

# Install gems
RUN bundle install --without development test

# Copy the main application.
COPY . .

# Start the application
CMD ["bundle", "exec", "rails", "server", "-b", "0.0.0.0"]
Worked fine before just slow
Brody
Brody•14mo ago
before you wernt using the private network, do you have a sleep 3 somewhere?
drdrigg
drdrigg•14mo ago
Just before I went to private. I tried sleep here:
No description
Brody
Brody•14mo ago
not quite how it works, remove that and do CMD sleep 3 && bundle exec rails server -b 0.0.0.0 in your dockerfile
drdrigg
drdrigg•14mo ago
Ok trying it out
drdrigg
drdrigg•14mo ago
Made no difference. I think one of the variables are wrong? The database name is wrong?
No description
Brody
Brody•14mo ago
the urls are correct, your code is doing something funky this time
char8
char8•14mo ago
I think DATABASE_URL on your Rails service is wrong? it looks like it thinks postgres.railway.internal is the database name, not the database host. Would also check your code and maybe add a logline to check that the connection string your app is using is what you expect it to be
drdrigg
drdrigg•14mo ago
Yes thinks so too, but which url should I use for internal?
Brody
Brody•14mo ago
please see this message
drdrigg
drdrigg•14mo ago
So set these in the postgres template? DATABASE_URL=postgres://${{PGUSER}}:${{POSTGRES_PASSWORD}}@${{RAILWAY_TCP_PROXY_DOMAIN}}:${{RAILWAY_TCP_PROXY_PORT}}/${{PGDATABASE}} DATABASE_PRIVATE_URL=postgres://${{PGUSER}}:${{POSTGRES_PASSWORD}}@${{RAILWAY_PRIVATE_DOMAIN}}:5432/${{PGDATABASE}} Adding/setting these now Deploying Its back to working now, but also the response time is 350 ms. database_url is now postgres://railway:[email protected]:5432/railway (I have changed a few characters to obfuscate the real url above) How can I see if it is using internal communication?
Brody
Brody•14mo ago
if you are using the internal domains, your using the private network, maybe sprinkle in some telemetry recording to measure RTT from your service to the databases? in my testing you can do a database connection and ping in 30-100ms (measured from within the apps code)
Want results from more Discord servers?
Add your server