EU servers are slower than US server (I am in the EU)
I have deployed an empty Postgres template, a Redis template and a Rails app in the European datacenter and it responds in 350 ms. But when I have the servers in the US West the same endpoint responds in 260 ms. I am located in Europe. How can the response be slower when using EU server (I am in the EU) than using US servers?
Project ID: 988851b6-78d5-44de-a73d-3e5b6468c8bf
78 Replies
Project ID:
988851b6-78d5-44de-a73d-3e5b6468c8bf@char8 I think you'd be the person to answer this 🙂
There’s a good chance this is fast pathing causing issues https://discord.com/channels/713503345364697088/1067670962276945961/1156374680937177098
I agree, but wouldn't hurt to hear char's opinion
Just figured I’d mention it here incase it was missed.
No this cannot be it. I have tried three independant ISPs (2 wifi and one celluar), same problem. Also I used AWS (which you use) many times before on both continents and never experienced this.
railway uses exclusively gcp at this time
Google Cloud Platform? The strange thing is that when using EU servers the response time should not be over 300 ms. I use the Vultr clouds for now (investigating if I can move this app to Railway) and the response time there is 111 ms for the same endpoint. Railway servers is 420 ms.
Could you check so that I have configured it correctly?
Also the railway database is almost empty (1 record) and the Vultr one is over 2 GB.
what's the hostname of your database
Is it this?
roundhouse.proxy.rlwy.net:11179
yep that's definitely using the new database templates that supports regions
Yes
I really want to use Railway for this project but four times slower is not ok unfortunately
and you've set the region with the drop-down and confirmed it deployed to the correct location by looking at the first log line of the build?
Yes I used the dropdown. Only the Rails app Build log contains anything, not the Postgres and Redis (but all dropdowns says EU servers).
if you make a bunch of requests, say 5-15 ish, do they all return at 350ms or or do you see a spread? I'm wondering if you're hitting the cache warming on the tcp proxies (the first couple of requests to each database will incur some high tail latencies)
Also can you list the callpath that's showing the 350ms? Eg: is it between you and your app or is it multi hop, eg: you <-> app <-> redis/postgres
I checked and everything is correctly deployed to the EU
Here is a few requests after each other on the Railway servers.
These are requests to the same codebase and endpoint and user but to my Vultr servers.
They are also located in Amsterdam
Also the Vultr servers are pretty heavy used and my Railway server only used by me.
yeah this is odd - I'm going to write up a quick benchmark app to double check and get back. You could try moving service <-> db communication over to private networks, those will bypass our TCP proxies and direct dial the db (so just 1 network hop).
How can I do that?
reference the
DATABASE_PRIVATE_URL insteadIn service variables?
So remove database_url and add DATABASE_PRIVATE_URL?
something like that yeah
There is no such option
Where can I find the value for DATABASE_PRIVATE_URL?
hmmm you may have deployed the template before the private variable was added
one sec
DATABASE_URL=${{Postgres.RAILWAY_PRIVATE_DOMAIN}}
use that on the service that connects to the databaseOk redeploying now
${{Postgres.RAILWAY_PRIVATE_DOMAIN}}
Saves as:
<Empty string>
can I see a screenshot of your project?
okay you did it a little differently than what I had in mind
DATABASE_URL=${{RAILWAY_PRIVATE_DOMAIN}}
try that insteadAdded like this (raw)
DATABASE_URL=${{RAILWAY_PRIVATE_DOMAIN}}
PGDATA=${{Postgres template.PGDATA}}
PGDATABASE=${{Postgres template.PGDATABASE}}
PGHOST=${{Postgres template.PGHOST}}
PGPASSWORD=${{Postgres template.PGPASSWORD}}
PGPORT=${{Postgres template.PGPORT}}
PGUSER=${{Postgres template.PGUSER}}
POSTGRES_DB=${{Postgres template.POSTGRES_DB}}
RAILS_LOG_TO_STDOUT=true
REDISHOST=${{Redis template.REDISHOST}}
REDISPASSWORD=${{Redis template.REDISPASSWORD}}
REDISPORT=${{Redis template.REDISPORT}}
REDISUSER=${{Redis template.REDISUSER}}
REDIS_PASSWORD=${{Redis template.REDIS_PASSWORD}}
REDIS_URL=${{Redis template.REDIS_URL}}
okay now you went back to the original way I wanted you to do it
so that's now wrong
in the service that Accesses the database
DATABASE_URL=${{Postgres.RAILWAY_PRIVATE_DOMAIN}}Changed, deploying
Got this now, the database_url is empty:
DATABASE_URL=${{Postgres.RAILWAY_PRIVATE_DOMAIN}}
PGDATA=${{Postgres template.PGDATA}}
PGDATABASE=${{Postgres template.PGDATABASE}}
PGHOST=${{Postgres template.PGHOST}}
PGPASSWORD=${{Postgres template.PGPASSWORD}}
PGPORT=${{Postgres template.PGPORT}}
PGUSER=${{Postgres template.PGUSER}}
POSTGRES_DB=${{Postgres template.POSTGRES_DB}}
REDISHOST=${{Redis template.REDISHOST}}
REDISPASSWORD=${{Redis template.REDISPASSWORD}}
REDISPORT=${{Redis template.REDISPORT}}
REDISUSER=${{Redis template.REDISUSER}}
REDIS_PASSWORD=${{Redis template.REDIS_PASSWORD}}
REDIS_URL=${{Redis template.REDIS_URL}}
Get a 503 error now
oh you've named your postgres database "Postgres template" please update the namespace in the variable I gave you
@drdrigg was it you that mentioned you’re testing all this on wifi? Do you have an ethernet connection available to test on?
it will give much more consistent results and eliminate many user error variables
What should I rename the template to?
the name is fine, I was just giving you a reference that uses the namespace of what the default name would be, you can just update the name in the reference to reflect your service name
What should I rename then?
DATABASE_URL=${{Postgres template.RAILWAY_PRIVATE_DOMAIN}}
see, you are using the namespace of the name of the postgres service, then referencing it's private networking domainTesting now
I assume the variable is rendering properly now?
Now PGhost and DATABASE_URL is the same
Not sure I am using the correct variables 🙂
Its a bit confusing
100% agree with you, this definitely can become very confusing, the team knows this and will be looking to improve their documentation
Should I remove all the variables? And if yes, which exactly should I add back?
hold on one sec, I screwed up, too early for me lol
set/overwrite these variables in your postgres service itself
in the service that accesses the postgres service
I know how this all works and I still messed it up
but those new variables should be correct
Server keeps crashing
With new variables
do they all render properly?
and can you please provide the error they fail with, I have an idea of why the connection fails
Ok will check
Well there is values in all when I click the eye icon
okay and let's see the full errors please
I tried to bundle exec db:create (to create the database since it said it was not created). Will try without this start command.
very not good error message lol
Its the gist of it haha
It cannot create the DB (not sure why it needs to do that since it was there before)
So I guess it has no connection to the DB
Get 503 now
can you add a 3 second delay in before the connection to the database attempt?
private networking takes some time to initialise
Where can I add that?
in code would probably be easiest
maybe even
sleep 3 && <your normal start command> in the settings menuWill try
On it
Made no difference
still error?
Must be an error somewhere with the database connection because my code outputs:
{"status":500,"error":"There was a server error","details":"FATAL: database "postgres.railway.internal" does not exist\n"}
are you building with a dockerfile?
Must be a variable issue (the system does not know what the db name is).
Yes
what image do you use
This is the content of my Dockerfile:
FROM ruby:2.6.8
Set environment variables
ENV RAILS_ROOT /app
ENV RAILS_ENV production
Set working directory
WORKDIR $RAILS_ROOT
Add Gemfile to the image
COPY Gemfile Gemfile.lock ./
Install gems
RUN bundle install --without development test
Copy the main application.
COPY . .
Start the application
CMD ["bundle", "exec", "rails", "server", "-b", "0.0.0.0"]
please enclose it in triple backticks
Worked fine before just slow
before you wernt using the private network, do you have a sleep 3 somewhere?
Just before I went to private. I tried sleep here:
not quite how it works, remove that and do
CMD sleep 3 && bundle exec rails server -b 0.0.0.0 in your dockerfileOk trying it out
Made no difference. I think one of the variables are wrong? The database name is wrong?
the urls are correct, your code is doing something funky this time
I think
DATABASE_URL on your Rails service is wrong? it looks like it thinks postgres.railway.internal is the database name, not the database host. Would also check your code and maybe add a logline to check that the connection string your app is using is what you expect it to beYes thinks so too, but which url should I use for internal?
please see this message
So set these in the postgres template?
DATABASE_URL=postgres://${{PGUSER}}:${{POSTGRES_PASSWORD}}@${{RAILWAY_TCP_PROXY_DOMAIN}}:${{RAILWAY_TCP_PROXY_PORT}}/${{PGDATABASE}}
DATABASE_PRIVATE_URL=postgres://${{PGUSER}}:${{POSTGRES_PASSWORD}}@${{RAILWAY_PRIVATE_DOMAIN}}:5432/${{PGDATABASE}}
Adding/setting these now
Deploying
Its back to working now, but also the response time is 350 ms.
database_url is now postgres://railway:-gF2fF2wCFg+eeCEFAbAaFgB3F56e-Ag-a@postgres.railway.internal:5432/railway
(I have changed a few characters to obfuscate the real url above)
How can I see if it is using internal communication?
if you are using the internal domains, your using the private network, maybe sprinkle in some telemetry recording to measure RTT from your service to the databases? in my testing you can do a database connection and ping in 30-100ms (measured from within the apps code)