Railway•8mo ago

Private Networking enables itself

I've discovered what I believe is weird and unintuitive behaviour. My app needs "Private Networking" to be disabled in order for the app to connect to my database(which is hosted outside of Railway, but it's the same for Railway-hosted db:s) but whenever I turn it off, it enables itself after a while. The interval seems to be like a day or so and it's something on Railway side that enables it. I'm unable to use Railway in this state because I can't have the time to enter the dashboard and disable it everytime it turns on.

Solution:

try setting the connect_timeout query parameter to 20

Jump to solution

16 Replies

Percy•8mo ago

Project ID: c39b75c1-5e52-4d8e-b559-4cc3d1c83383

misimilen•8mo ago

c39b75c1-5e52-4d8e-b559-4cc3d1c83383

Brody•8mo ago

you can always increase the connection timeout for your database or add a sleep for 2 seconds before you try to connect to the database

misimilen•8mo ago

I'm not sure it's the timeout because when it fails to reach the db, it immediately fails. Here it tries and tries and tries and finally it succeeds. It's something else going on which I can't debug because I don't know how the private networking is done in Railway.

Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "xxxxxx", schema "public" at "xxxxxx"
Error: P1001: Can't reach database server at `xxxxxx`:`5432`
Please make sure your database server is running at `xxxxxx`:`5432`.
npm WARN config production Use `--omit=dev` instead.
> monorepo@0.0.0 prisma:deploy
> cd packages/backend && npx prisma migrate deploy
npm WARN config production Use `--omit=dev` instead.
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "xxxxxx", schema "public" at "xxxxxx"
Error: P1001: Can't reach database server at `xxxxxx`:`5432`
Please make sure your database server is running at `xxxxxx`:`5432`.
npm WARN config production Use `--omit=dev` instead.
> monorepo@0.0.0 prisma:deploy
> cd packages/backend && npx prisma migrate deploy
npm WARN config production Use `--omit=dev` instead.
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "xxxxxx", schema "public" at "xxxxxx"
Error: P1001: Can't reach database server at `xxxxxx`:`5432`
Please make sure your database server is running at `xxxxxx`:`5432`.
npm WARN config production Use `--omit=dev` instead.
> monorepo@0.0.0 prisma:deploy
> cd packages/backend && npx prisma migrate deploy
npm WARN config production Use `--omit=dev` instead.
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "xxxxxx", schema "public" at "xxxxxx"
10 migrations found in prisma/migrations
No pending migrations to apply.
npm WARN config production Use `--omit=dev` instead.
> monorepo@0.0.0 start:backend
> cd packages/backend && node ./build/index.js
[13:34:45.351] INFO (156): Server listening at http://0.0.0.0:6231

Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "xxxxxx", schema "public" at "xxxxxx"
Error: P1001: Can't reach database server at `xxxxxx`:`5432`
Please make sure your database server is running at `xxxxxx`:`5432`.
npm WARN config production Use `--omit=dev` instead.
> monorepo@0.0.0 prisma:deploy
> cd packages/backend && npx prisma migrate deploy
npm WARN config production Use `--omit=dev` instead.
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "xxxxxx", schema "public" at "xxxxxx"
Error: P1001: Can't reach database server at `xxxxxx`:`5432`
Please make sure your database server is running at `xxxxxx`:`5432`.
npm WARN config production Use `--omit=dev` instead.
> monorepo@0.0.0 prisma:deploy
> cd packages/backend && npx prisma migrate deploy
npm WARN config production Use `--omit=dev` instead.
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "xxxxxx", schema "public" at "xxxxxx"
Error: P1001: Can't reach database server at `xxxxxx`:`5432`
Please make sure your database server is running at `xxxxxx`:`5432`.
npm WARN config production Use `--omit=dev` instead.
> monorepo@0.0.0 prisma:deploy
> cd packages/backend && npx prisma migrate deploy
npm WARN config production Use `--omit=dev` instead.
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "xxxxxx", schema "public" at "xxxxxx"
10 migrations found in prisma/migrations
No pending migrations to apply.
npm WARN config production Use `--omit=dev` instead.
> monorepo@0.0.0 start:backend
> cd packages/backend && node ./build/index.js
[13:34:45.351] INFO (156): Server listening at http://0.0.0.0:6231

I believe the private networking should remain off until I enable it myself. Ah, the timestamps are missing from the logs I posted... but it explains it doesn't wait for the timeout.

Brody•8mo ago

yes it coming back on is a bug for sure, but that's not something we can fix right now, so let's get your code to work around it with prisma you are using env("DATABASE_URL") right?

misimilen•8mo ago

Yes, I've added the pool_timeout and set it to 20 seconds but I'm not sure it's used for when Prisma does the migration. postgres://xxxxxxxx:xxxxxxxxx@xxxxx/xxxxxx?connection_limit=4&pool_timeout=20 I guess I can just crank up the On-Failure Restart Policy Max Retry to like a 100 or something 😄

Solution

Brody•8mo ago

try setting the connect_timeout query parameter to 20

Brody•8mo ago

please don't rely on a crashing app for database connection retry attempts https://www.prisma.io/docs/concepts/database-connectors/postgresql

misimilen•8mo ago

desperate times calls for desperate measures 😄 but I will try that parameter now I set it to 30 just to be sure

Brody•8mo ago

I have seen 20 make it work

misimilen•8mo ago

That did it I think!

Brody•8mo ago

awesome

misimilen•8mo ago

I posted a bug report on the https://feedback.railway.app/ so I'll close this thread now

Railway Feedback

Give feedback to the Railway team so we can make more informed product decisions. Powered by Canny.

Brody•8mo ago

oh they are very aware of this already if there was no private network initialisation times then you would never need to disable it in the first place because then it would be totally invisible to the user

misimilen•8mo ago

I guess but I'm so paranoid with all types of networking layers and such because I rather not have it so that I can more easily know where the slowness in my app comes from.

Brody•8mo ago

yeah I get ya, but if there was no initialisation times, then there truly would be no difference to the user experience between having it on and having it off it's just the network connections that are made within the first 2 seconds of app start that get messed up, after that there's no issues. and in fact you could have slapped a sleep 2 && in front of your start command and called it a day too (just make sure you use a healthcheck in that case (you should always use a healthcheck))

Gaming

Programming

Private Networking enables itself