Railway•9mo ago

railway internal network is unreadable

Solution:

try setting the ENABLE_ALPINE_PRIVATE_NETWORKING=true envvar per the docs on all the alpine containers?

Jump to solution

26 Replies

Percy•9mo ago

Project ID: 1f736e16-6743-4b65-ac36-e7b8aca242da

waltcow•9mo ago

projectID : 1f736e16-6743-4b65-ac36-e7b8aca242da

Brody•9mo ago

is that the correct domain name for the service?

waltcow•9mo ago

Brody•9mo ago

at what point in your apps lifecycle do you make a call this this internal service

waltcow•9mo ago

exactly the internal domain i use frontend web framework nextjs, and proxy api/* the backend internal sever call 'dify-flask'

waltcow•9mo ago

Brody•9mo ago

I'm sorry but that doesn't quite answer the question

waltcow•9mo ago

it use nodejs proxy to backend api server log show dns lookup fail

Brody•9mo ago

at what point in your apps lifecycle do you make a call this this internal service

waltcow•9mo ago

when browser call http://exapmle.com/api/foo/bar , nodejs proxy to backend http://dify-flask.railway.internal:5001/foo/bar

Brody•9mo ago

okay thank you has this exact code worked in the past?

waltcow•9mo ago

yes

Brody•9mo ago

have you tried a redoployment?

waltcow•9mo ago

redoployment twice . both frontend and backend

Brody•9mo ago

@fp could there be a problem with private network resolution?

Faraz•9mo ago

Interesting. Let me raise internally and find out.

char8•9mo ago

Is the container using alpine by chance? Or musl libc? The DNS resolution for that is slightly different and we have a note in the docs about it: https://docs.railway.app/reference/private-networking#workaround-for-alpine-based-images (We're working on a fix for that particular problem)

waltcow•9mo ago

FROM node:18.17.0-alpine

Solution

char8•9mo ago

try setting the ENABLE_ALPINE_PRIVATE_NETWORKING=true envvar per the docs on all the alpine containers?

waltcow•9mo ago

OK, i try it now

char8•9mo ago

to give some background, we put both our internal DNS server and a public DNS resolver in resolv.conf . The dns lookup functions in glibc will treat the second entry as a fallback (desired behaviour since we want public DNS to work even if our internal resolver is unavailable), MUSL libc (used by alpine) is different: it sends out parallel requests to all resolvers in resolv.conf and errors on the first failure. Because public DNS doesn't have the railway.internal zone, privnet lookups can fail on Alpine.

waltcow•9mo ago

it works now

char8•9mo ago

what the magic envvar above does is it removes the fallback resolv.conf entry, the only caveat with this is that it means public DNS resolutions are only available with privnet initialization (which is usually ~ 100ms after container startup). Again this sucks, so we're actively playing with a fix for it

waltcow•9mo ago

network issue occur again today

at connResetException (node:internal/errors:720:14)

at Socket.socketCloseListener (node:_http_client:474:25)

at Socket.emit (node:events:526:35)

at TCP.<anonymous> (node:net:323:12)

at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {

code: 'ECONNRESET'

}

Error: socket hang up

at connResetException (node:internal/errors:720:14)

at Socket.socketCloseListener (node:_http_client:474:25)

at Socket.emit (node:events:526:35)

at TCP.<anonymous> (node:net:323:12)

at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {

code: 'ECONNRESET'

}

at connResetException (node:internal/errors:720:14)

at Socket.socketCloseListener (node:_http_client:474:25)

at Socket.emit (node:events:526:35)

at TCP.<anonymous> (node:net:323:12)

at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {

code: 'ECONNRESET'

}

Error: socket hang up

at connResetException (node:internal/errors:720:14)

at Socket.socketCloseListener (node:_http_client:474:25)

at Socket.emit (node:events:526:35)

at TCP.<anonymous> (node:net:323:12)

at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {

code: 'ECONNRESET'

}

@char8

Brody•9mo ago

do not ping the team #🛂｜readme #5

Gaming

Programming

railway internal network is unreadable