Is private networking down?

Hey is private networking down or something? Because currently all my builds are crashing because they can't communicate through private networking, host names don't resolve
19 Replies
Percy
Percy11mo ago
Project ID: N/A
Brody
Brody11mo ago
AFAIK the private network isn't available during build
Andrew Bekhiet
Andrew Bekhiet11mo ago
It's not during build, it's after the build, when the app is running
Brody
Brody11mo ago
oh you said builds what errors are you getting
Andrew Bekhiet
Andrew Bekhiet11mo ago
Oh sorry I meant my services are crashing Project ID: 152941be-0212-4cbc-8977-c7d8fa719760 stderr: 'pg_dump: error: connection to database "(database-nane)" failed: could not translate host name "(service-name).railway.internal" to address: Name does not resolve\n' } And also hasura API service is failing with a similar error Error from hasura: connection error","internal":"could not translate host name "(service-name)" to address: Name or service not known
Brody
Brody11mo ago
I think I know the issue, private network initialisation takes about 2 seconds on average after container start, this can also effect calls to external APIs, add a timeout or increase DNS resolution time or add a retry mechanism
Andrew Bekhiet
Andrew Bekhiet11mo ago
Hasura has a retry mechanism and it still fails It was working about half an hour ago, but now after redeploying all services, they can't communicate through private networking
Brody
Brody11mo ago
oh well thats not ideal redeploy your services again? if the problem persists i will flag the team
Andrew Bekhiet
Andrew Bekhiet11mo ago
Sadly, the problem still persists
Duchess
Duchess11mo ago
Thread has been flagged to Railway team by @Brody.
char8
char811mo ago
did you rename the services by chance? I see some lookups with '-' in the service name, but the service names have '_' in them.
Andrew Bekhiet
Andrew Bekhiet11mo ago
Yes I did
char8
char811mo ago
I would advise renaming everything to have - in them, going to track down our validation logic - but _ definitely will cause issues in dns names, so we're probably missing some validation to prevent it the service name is used to create the dns name, I think we had a warning when you try to rename the service - will follow that up as well
Andrew Bekhiet
Andrew Bekhiet11mo ago
Currently only the service name has but the private networking says its hostname containing - If I renamed the service name to contain would it work?
char8
char811mo ago
if you could just rename the private networking endpoints to all have dashes - and no underscores, and then reference stuff using that DNS name, that should work. There's an edit button on hover. Did you rename both the endpoint and the service or just the service?
No description
char8
char811mo ago
trying to work out how we got the service into this state since we do validate against underscores 😅
Andrew Bekhiet
Andrew Bekhiet11mo ago
While renaming the endpoint now I discovered it had Reverting back to - makes it work Sorry for this silly mistake and thank you so much❤️❤️❤️ Yeah I noticed this the first time I renamed the endpoint but I don't remember that I renamed the endpoint to have again
Brody
Brody11mo ago
i would have never had guessed that
char8
char811mo ago
no worries at all - We should definitely defend against that, going to to ticket internally to root cause. Cos it doesn't let me rename stuff with an underscore.
No description