Coder.com•2d ago

failed to WebSocket dial: failed to send handshake request on private EKS

I have a private EKS cluster with cert-manager and DNS installed on the cluster. Attached are my test files. I exec inside the pod with a kubectl exec command. Once I'm inside the pod I log into coder with the coder login command. From there I pull, modify, and push my template. When I run the coder template push command I get the following error: coder template push -y === ✔ Queued [0ms] ==> ⧗ Running ==> ⧗ Running Encountered an error running "coder templates push", see "coder templates push --help" for more information error: API request error to "GET:http://coder.example.com/api/v2/templateversions/90463ed1-d72f-46bf-a4eb-e9f8f8d3d139/logs?follow". Status code 426 Trace=[begin streaming logs: ] unexpected non-JSON response "text/plain; charset=utf-8" WebSocket protocol violation: Connection header "close" does not contain Upgrade {"message":"Failed to accept websocket.","detail":"failed to accept WebSocket connection: WebSocket protocol violation: Connection header "close" does not contain Upgrade"} ``` This error also occurs when I'm not inside a pod. When I login to coder via CLI from outside the cluster, I get the same error.

cert.yaml

coder-BETA-VALUES.ya...

Dockerfile

15 Replies

Phorcys•22h ago

how's your healthcheck page looking?

yuji.fushiguroOP•22h ago

The healthcheck page is showing the same error for Websocket with error code EWS01

yuji.fushiguroOP•22h ago

I did find a work around to be able to push my templates from my local machine and I was able to do so by using port forwarding. It's not ideal but it helped me get to the next step.

k port-forward svc/coder 8080:80 -n coder
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

k port-forward svc/coder 8080:80 -n coder
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

I still would like to get pass this websocket issuse. This helped me be able to push templates, but then I ran into the same websocket error when my workspace tries to spin up. Here are the logs from the pod that's trying to start the workspace

message.txt

Phorcys•22h ago

yeah this makes sense so this means that your ingress controller is not allowing the upgrades do you only have an ingress in front of the Coder service or do you also have a load balancer of some kind?

yuji.fushiguroOP•22h ago

I have a loadbalancer. I did add a code snippet to my ingress to my ingress as a troubleshooting approach but don't know if I added correctly

ingress:
    enable: true
    className: nginx
    host: coder.example.com
    wildcardHost: "*.example.com"
    annotations:
      {
        cert-manager.io/cluster-issuer: coder-issuer,
        cert-manager.io/issuer-kind: Issuer,
        nginx.ingress.kubernetes.io/server-snippets: 
          "proxy_set_header Upgrade $http_upgrade;
          proxy_http_version 1.1;
          proxy_set_header X-Forwarded-Host $http_host;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_set_header X-Forwarded-For $remote_addr;
          proxy_set_header Host $host;
          proxy_set_header Connection "upgrade";
          proxy_cache_bypass $http_upgrade;"
      }

ingress:
    enable: true
    className: nginx
    host: coder.example.com
    wildcardHost: "*.example.com"
    annotations:
      {
        cert-manager.io/cluster-issuer: coder-issuer,
        cert-manager.io/issuer-kind: Issuer,
        nginx.ingress.kubernetes.io/server-snippets: 
          "proxy_set_header Upgrade $http_upgrade;
          proxy_http_version 1.1;
          proxy_set_header X-Forwarded-Host $http_host;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_set_header X-Forwarded-For $remote_addr;
          proxy_set_header Host $host;
          proxy_set_header Connection "upgrade";
          proxy_cache_bypass $http_upgrade;"
      }

Phorcys•22h ago

i'm pretty sure it's enabled by default with nginx as i'm using it and didn't have to do anthing specific what LB are you using?

yuji.fushiguroOP•22h ago

By LB, do you mean Network or Application? I have a Network Load Balancer

Phorcys•22h ago

i just meant load balancer in general oh wait so it's only a network-level one that doesn't handle http traffic in any specific way?

yuji.fushiguroOP•22h ago

Only through port base routing on port 80 and 443 to my nginx-ingress controller target groups

Phorcys•22h ago

ah okay yeah i thought you coudl've been using AWS ALB or something similar

Support for websockets is provided by NGINX out of the box. No special configuration required.

https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#websockets yeah so i'm honestly unsure, but something is messing with your upgrades are you using an http proxy on the client device?

yuji.fushiguroOP•22h ago

I can ask our network guy if websockets are be block by default but I'm unsure. I do know we are connect to our network via VPN, but any other networking stuff is beyond me 🤯 *being

Phorcys•22h ago

yeah maybe your ingress controller was reconfigured to not allow it by default?

yuji.fushiguroOP•21h ago

I'm not seeing anything within the helm values for the loadbalancer or ingress controller that would suggest otherwise. I did ping our networking guy so I'll wait to see what he says Network guy has confirmed that websocket traffic is being blocked

Phorcys•21h ago

ah yeah that makes more sense! yeah so you'll need to get it unblocked somehow

yuji.fushiguroOP•21h ago

I have a request in that will go in tonight and we'll see how it goes. I do have one more problem I would like to resolve. I can't access my dashboard via HTTPS even though I have a certifacte attached to my deployment. From looking at my example files do you know why that might be?

Gaming

Programming

failed to WebSocket dial: failed to send handshake request on private EKS

Did you find this page helpful?