R
Railwayβ€’13mo ago
misimilen

Private Networking enables itself

I've discovered what I believe is weird and unintuitive behaviour. My app needs "Private Networking" to be disabled in order for the app to connect to my database(which is hosted outside of Railway, but it's the same for Railway-hosted db:s) but whenever I turn it off, it enables itself after a while. The interval seems to be like a day or so and it's something on Railway side that enables it. I'm unable to use Railway in this state because I can't have the time to enter the dashboard and disable it everytime it turns on.
No description
Solution:
try setting the connect_timeout query parameter to 20
Jump to solution
16 Replies
Percy
Percyβ€’13mo ago
Project ID: c39b75c1-5e52-4d8e-b559-4cc3d1c83383
misimilen
misimilenβ€’13mo ago
c39b75c1-5e52-4d8e-b559-4cc3d1c83383
Brody
Brodyβ€’13mo ago
you can always increase the connection timeout for your database or add a sleep for 2 seconds before you try to connect to the database
misimilen
misimilenβ€’13mo ago
I'm not sure it's the timeout because when it fails to reach the db, it immediately fails. Here it tries and tries and tries and finally it succeeds. It's something else going on which I can't debug because I don't know how the private networking is done in Railway.
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "xxxxxx", schema "public" at "xxxxxx"
Error: P1001: Can't reach database server at `xxxxxx`:`5432`
Please make sure your database server is running at `xxxxxx`:`5432`.
npm WARN config production Use `--omit=dev` instead.
> [email protected] prisma:deploy
> cd packages/backend && npx prisma migrate deploy
npm WARN config production Use `--omit=dev` instead.
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "xxxxxx", schema "public" at "xxxxxx"
Error: P1001: Can't reach database server at `xxxxxx`:`5432`
Please make sure your database server is running at `xxxxxx`:`5432`.
npm WARN config production Use `--omit=dev` instead.
> [email protected] prisma:deploy
> cd packages/backend && npx prisma migrate deploy
npm WARN config production Use `--omit=dev` instead.
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "xxxxxx", schema "public" at "xxxxxx"
Error: P1001: Can't reach database server at `xxxxxx`:`5432`
Please make sure your database server is running at `xxxxxx`:`5432`.
npm WARN config production Use `--omit=dev` instead.
> [email protected] prisma:deploy
> cd packages/backend && npx prisma migrate deploy
npm WARN config production Use `--omit=dev` instead.
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "xxxxxx", schema "public" at "xxxxxx"
10 migrations found in prisma/migrations
No pending migrations to apply.
npm WARN config production Use `--omit=dev` instead.
> [email protected] start:backend
> cd packages/backend && node ./build/index.js
[13:34:45.351] INFO (156): Server listening at http://0.0.0.0:6231
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "xxxxxx", schema "public" at "xxxxxx"
Error: P1001: Can't reach database server at `xxxxxx`:`5432`
Please make sure your database server is running at `xxxxxx`:`5432`.
npm WARN config production Use `--omit=dev` instead.
> [email protected] prisma:deploy
> cd packages/backend && npx prisma migrate deploy
npm WARN config production Use `--omit=dev` instead.
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "xxxxxx", schema "public" at "xxxxxx"
Error: P1001: Can't reach database server at `xxxxxx`:`5432`
Please make sure your database server is running at `xxxxxx`:`5432`.
npm WARN config production Use `--omit=dev` instead.
> [email protected] prisma:deploy
> cd packages/backend && npx prisma migrate deploy
npm WARN config production Use `--omit=dev` instead.
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "xxxxxx", schema "public" at "xxxxxx"
Error: P1001: Can't reach database server at `xxxxxx`:`5432`
Please make sure your database server is running at `xxxxxx`:`5432`.
npm WARN config production Use `--omit=dev` instead.
> [email protected] prisma:deploy
> cd packages/backend && npx prisma migrate deploy
npm WARN config production Use `--omit=dev` instead.
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "xxxxxx", schema "public" at "xxxxxx"
10 migrations found in prisma/migrations
No pending migrations to apply.
npm WARN config production Use `--omit=dev` instead.
> [email protected] start:backend
> cd packages/backend && node ./build/index.js
[13:34:45.351] INFO (156): Server listening at http://0.0.0.0:6231
I believe the private networking should remain off until I enable it myself. Ah, the timestamps are missing from the logs I posted... but it explains it doesn't wait for the timeout.
Brody
Brodyβ€’13mo ago
yes it coming back on is a bug for sure, but that's not something we can fix right now, so let's get your code to work around it with prisma you are using env("DATABASE_URL") right?
misimilen
misimilenβ€’13mo ago
Yes, I've added the pool_timeout and set it to 20 seconds but I'm not sure it's used for when Prisma does the migration. postgres://xxxxxxxx:xxxxxxxxx@xxxxx/xxxxxx?connection_limit=4&pool_timeout=20 I guess I can just crank up the On-Failure Restart Policy Max Retry to like a 100 or something πŸ˜„
Solution
Brody
Brodyβ€’13mo ago
try setting the connect_timeout query parameter to 20
Brody
Brodyβ€’13mo ago
please don't rely on a crashing app for database connection retry attempts https://www.prisma.io/docs/concepts/database-connectors/postgresql
misimilen
misimilenβ€’13mo ago
desperate times calls for desperate measures πŸ˜„ but I will try that parameter now I set it to 30 just to be sure
Brody
Brodyβ€’13mo ago
I have seen 20 make it work
misimilen
misimilenβ€’13mo ago
That did it I think!
Brody
Brodyβ€’13mo ago
awesome
misimilen
misimilenβ€’13mo ago
I posted a bug report on the https://feedback.railway.app/ so I'll close this thread now
Railway Feedback
Give feedback to the Railway team so we can make more informed product decisions. Powered by Canny.
Brody
Brodyβ€’13mo ago
oh they are very aware of this already if there was no private network initialisation times then you would never need to disable it in the first place because then it would be totally invisible to the user
misimilen
misimilenβ€’13mo ago
I guess but I'm so paranoid with all types of networking layers and such because I rather not have it so that I can more easily know where the slowness in my app comes from.
Brody
Brodyβ€’13mo ago
yeah I get ya, but if there was no initialisation times, then there truly would be no difference to the user experience between having it on and having it off it's just the network connections that are made within the first 2 seconds of app start that get messed up, after that there's no issues. and in fact you could have slapped a sleep 2 && in front of your start command and called it a day too (just make sure you use a healthcheck in that case (you should always use a healthcheck))
Want results from more Discord servers?
Add your server