Coder.com•6mo ago

failed to WebSocket dial: failed to send handshake request on private EKS

I have a private EKS cluster with cert-manager and DNS installed on the cluster. Attached are my test files. I exec inside the pod with a kubectl exec command. Once I'm inside the pod I log into coder with the coder login command. From there I pull, modify, and push my template. When I run the coder template push command I get the following error: coder template push -y === ✔ Queued [0ms] ==> ⧗ Running ==> ⧗ Running Encountered an error running "coder templates push", see "coder templates push --help" for more information error: API request error to "GET:http://coder.example.com/api/v2/templateversions/90463ed1-d72f-46bf-a4eb-e9f8f8d3d139/logs?follow". Status code 426 Trace=[begin streaming logs: ] unexpected non-JSON response "text/plain; charset=utf-8" WebSocket protocol violation: Connection header "close" does not contain Upgrade {"message":"Failed to accept websocket.","detail":"failed to accept WebSocket connection: WebSocket protocol violation: Connection header "close" does not contain Upgrade"} ``` This error also occurs when I'm not inside a pod. When I login to coder via CLI from outside the cluster, I get the same error.

cert.yaml

coder-BETA-VALUES.ya...

Dockerfile

Solution:

Hey @Phorcys I fixed the websocket error. Within my ingress annotations I had there following nginx.ingress.kubernetes.io/websocket-service: “coder”...

Jump to solution

45 Replies

Phorcys•6mo ago

how's your healthcheck page looking?

yuji.fushiguroOP•6mo ago

The healthcheck page is showing the same error for Websocket with error code EWS01

yuji.fushiguroOP•6mo ago

I did find a work around to be able to push my templates from my local machine and I was able to do so by using port forwarding. It's not ideal but it helped me get to the next step.

k port-forward svc/coder 8080:80 -n coder
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

k port-forward svc/coder 8080:80 -n coder
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

I still would like to get pass this websocket issuse. This helped me be able to push templates, but then I ran into the same websocket error when my workspace tries to spin up. Here are the logs from the pod that's trying to start the workspace

message.txt

Phorcys•6mo ago

yeah this makes sense so this means that your ingress controller is not allowing the upgrades do you only have an ingress in front of the Coder service or do you also have a load balancer of some kind?

yuji.fushiguroOP•6mo ago

I have a loadbalancer. I did add a code snippet to my ingress to my ingress as a troubleshooting approach but don't know if I added correctly

ingress:
    enable: true
    className: nginx
    host: coder.example.com
    wildcardHost: "*.example.com"
    annotations:
      {
        cert-manager.io/cluster-issuer: coder-issuer,
        cert-manager.io/issuer-kind: Issuer,
        nginx.ingress.kubernetes.io/server-snippets: 
          "proxy_set_header Upgrade $http_upgrade;
          proxy_http_version 1.1;
          proxy_set_header X-Forwarded-Host $http_host;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_set_header X-Forwarded-For $remote_addr;
          proxy_set_header Host $host;
          proxy_set_header Connection "upgrade";
          proxy_cache_bypass $http_upgrade;"
      }

ingress:
    enable: true
    className: nginx
    host: coder.example.com
    wildcardHost: "*.example.com"
    annotations:
      {
        cert-manager.io/cluster-issuer: coder-issuer,
        cert-manager.io/issuer-kind: Issuer,
        nginx.ingress.kubernetes.io/server-snippets: 
          "proxy_set_header Upgrade $http_upgrade;
          proxy_http_version 1.1;
          proxy_set_header X-Forwarded-Host $http_host;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_set_header X-Forwarded-For $remote_addr;
          proxy_set_header Host $host;
          proxy_set_header Connection "upgrade";
          proxy_cache_bypass $http_upgrade;"
      }

Phorcys•6mo ago

i'm pretty sure it's enabled by default with nginx as i'm using it and didn't have to do anthing specific what LB are you using?

yuji.fushiguroOP•6mo ago

By LB, do you mean Network or Application? I have a Network Load Balancer

Phorcys•6mo ago

i just meant load balancer in general oh wait so it's only a network-level one that doesn't handle http traffic in any specific way?

yuji.fushiguroOP•6mo ago

Only through port base routing on port 80 and 443 to my nginx-ingress controller target groups

Phorcys•6mo ago

ah okay yeah i thought you coudl've been using AWS ALB or something similar

Support for websockets is provided by NGINX out of the box. No special configuration required.

https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#websockets yeah so i'm honestly unsure, but something is messing with your upgrades are you using an http proxy on the client device?

yuji.fushiguroOP•6mo ago

I can ask our network guy if websockets are be block by default but I'm unsure. I do know we are connect to our network via VPN, but any other networking stuff is beyond me 🤯 *being

Phorcys•6mo ago

yeah maybe your ingress controller was reconfigured to not allow it by default?

yuji.fushiguroOP•6mo ago

I'm not seeing anything within the helm values for the loadbalancer or ingress controller that would suggest otherwise. I did ping our networking guy so I'll wait to see what he says Network guy has confirmed that websocket traffic is being blocked

Phorcys•6mo ago

ah yeah that makes more sense! yeah so you'll need to get it unblocked somehow

yuji.fushiguroOP•6mo ago

I have a request in that will go in tonight and we'll see how it goes. I do have one more problem I would like to resolve. I can't access my dashboard via HTTPS even though I have a certifacte attached to my deployment. From looking at my example files do you know why that might be? Update: Firewall rule has been updated and confirmed by network guy that it is being allowed in the logs, but error is still persisting.

Phorcys•5mo ago

hey! sorry for the delay! were you able to solve this?

yuji.fushiguroOP•5mo ago

The WebSocket issue? No. The TLS access URL? Yes.

Phorcys•5mo ago

yeah I meant the websocket issue i'm not sure how to debug this given that websocket traffic has to be blocked or unsupported somewhere still what if you remove that nginx.ingress.kubernetes.io/server-snippets part?

yuji.fushiguroOP•5mo ago

The error persisted. As far as the docs for Kubernetes TLS configuration, do you need a wildcard secret name? Because wildcards are blocked from being created for me

Phorcys•5mo ago

you do only if you enable the wildcard subdomain and want Coder to have access to the cert in general you'd let the ingress take care of that anyways but in your case it hardly matters sorry @yuji.fushiguro, I am not sure how to help as it seems that your reverse-proxy setup is somehow denying the upgrades and I'm not sure what could be causing this

Solution

yuji.fushiguro•5mo ago

Hey @Phorcys I fixed the websocket error. Within my ingress annotations I had there following nginx.ingress.kubernetes.io/websocket-service: “coder” I just updated nginx.org/websocket-service: “coder” And everything started working. Now I just gotta figure out how to get coder web browser to use IPv4 instead of IPv6. I’m getting a 502 error and I try to open code-server in the browser. But I’m able to use VS Code Desktop just fine.

yuji.fushiguroOP•5mo ago

I'm not understanding why I'm getting this error when I try to access the code-server in the web browser from my workspace. I've added NO_PROXY=::/0 to my environment variables but it still shows an IPv6 number on port 13337.

yuji.fushiguroOP•5mo ago

Phorcys•5mo ago

great! thanks for following-up!

Phorcys•5mo ago

this is a local IP in your cluster and isn't related to your proxy settings VSCode Desktop is a link that doesn't need a port and only relies on your Coder agent being healthy, so it makes sense that it could be working when code-server isn't would you mind sharing your Terraform template? also, does curl http://localhost:13337 in the workspace work?

yuji.fushiguroOP•5mo ago

I'm using the default kubernetes-devcontainer template https://github.com/coder/coder/tree/main/examples/templates/kubernetes-devcontainer

GitHub

coder/examples/templates/kubernetes-devcontainer at main · coder/c...

Provision remote development environments via Terraform - coder/coder

yuji.fushiguroOP•5mo ago

These are the logs from the workspace spinning up

Phorcys•5mo ago

ah yeah totally makes sense that it doesn't work then

curl: (35) Recv failure: Connection reset by peer
🥳 code-server has been installed in /tmp/code-server

bash: line 72: /tmp/code-server/bin/code-server: No such file or directory

curl: (35) Recv failure: Connection reset by peer
🥳 code-server has been installed in /tmp/code-server

bash: line 72: /tmp/code-server/bin/code-server: No such file or directory

yuji.fushiguroOP•5mo ago

message.txt

yuji.fushiguroOP•5mo ago

Yea, I've seen that and don't understand why it's not there. I try adding something to the startup script but it still doesn't work. I've exec'd into the workspace pod and was able to curl http://coder.example.com/bin/coder-linux-amd64 just fine. Also within the workspace pod, the coder binary is in the directory /tmp/coder.Dkjdacli/coder

Phorcys•5mo ago

my guess is that your proxy is (not) doing it's job (inside the workspace)

Also within the workspace pod, the coder binary is in the directory /tmp/coder.Dkjdacli/coder

that's fine though, the location of the coder binary is unrelated to where the module DLs stuff

yuji.fushiguroOP•5mo ago

I was wondering if there was something I can do to the template first before creating my own from scratch. I wanted to take the easier approach.

Phorcys•5mo ago

oh yeah definitely, you usually don't want to be creating them from scratch at all i'll reply in a moment, just working on some stuff atm what's the output of curl -fsSL https://code-server.dev/install.sh | sh? also, does it work if you do export 'NO_PROXY=::/0'; curl -fsSL https://code-server.dev/install.sh | sh

yuji.fushiguroOP•5mo ago

Am I curling this from my local machine or inside the pod?

Phorcys•5mo ago

inside the workspace this one

yuji.fushiguroOP•5mo ago

root@coder-a3052628-3b3f-48fe-82ea-661fdd7062fc-89d9dc69-z4c6f:/# curl -fsSL https://code-server.dev/install.sh | sh curl: (35) Recv failure: Connection reset by peer root@coder-a3052628-3b3f-48fe-82ea-661fdd7062fc-89d9dc69-z4c6f:/# curl -fsSL http://coder.example.com/install.sh | sh Installing coder-linux-amd64 v2.19.0+2f32b11 from http://coder.example.com. + mkdir -p ~/.cache/coder/local_downloads + curl -#fL -o ~/.cache/coder/local_downloads/coder-linux-amd64-v2.19.0+2f32b11.incomplete -C - http://coder.example.com/bin/coder-linux-amd64 ######################################################################## 100.0% + mv ~/.cache/coder/local_downloads/coder-linux-amd64-v2.19.0+2f32b11.incomplete ~/.cache/coder/local_downloads/coder-linux-amd64-v2.19.0+2f32b11 + mkdir -p /usr/local + mkdir -p /usr/local/bin + cp ~/.cache/coder/local_downloads/coder-linux-amd64-v2.19.0+2f32b11 /usr/local/bin/coder + chmod +x /usr/local/bin/coder Coder v2.19.0+2f32b11 installed. The Coder binary has been placed in the following location: /usr/local/bin/coder To run a Coder server: $ coder server To connect to a Coder deployment: $ coder login <deployment url> root@coder-a3052628-3b3f-48fe-82ea-661fdd7062fc-89d9dc69-z4c6f:/# If I use my access URL I'm able to cURL the install script Since my cluster doesn't have access to the internet

Phorcys•5mo ago

hey! this is normal code-server is a different thing altogether the reason this is not working is because your template has the code-server module (https://registry.coder.com/modules/coder/code-server) in it, which tries to download code-server from the outside so if you don't want it you can just remove it, or download code-server in the image itself

yuji.fushiguroOP•5mo ago

Okay makes sense. So code-server can’t be installed from my coder pod? I know I can download the coder CLI, but can I also download the server as well? Or I have to bake it into the image like you said?

Phorcys•5mo ago

i think there's a misunderstanding

Phorcys•5mo ago

code-server is a separate thing https://github.com/coder/code-server

GitHub

GitHub - coder/code-server: VS Code in the browser

VS Code in the browser. Contribute to coder/code-server development by creating an account on GitHub.

Phorcys•5mo ago

it's just VS Code in the browser and not needed to run Coder/not related with Coder as a product

yuji.fushiguroOP•5mo ago

Yes. This is what I was trying to clarify. Thank you.

Phorcys•5mo ago