DERP requires connection upgrade

Hi,

My company tasked me with finding a solution for setting up a remote development environment, so here I am.
As our own proof of concept, we have an AWS EC2 instance running and on it we have a "coder" container.
Coder service and workspaces run fine and dandy if we use the public IP of the instance as CODER_ACCESS_URL value.

Replacing the CODER_ACCESS_URL with our domain (ie coder.company.com), is where we hit the wall.

Coder dashboard works, I can create templates and from them workspaces, however, connection to a workspace does not work.

Looking at the code server logs, this is the message that is shown:

 {
"host": "coder.company.com", 
"path": "/derp", 
"proto": "HTTP/1.1", 
"remote_addr": "172.20.0.4:54062", 
"took": "69.837µs", 
"status_code": 426, 
"latency_ms": 0, 
"response_body": "DERP requires connection upgrade\n", 
"request_id": "143c2491-55dc-4e83-8d69-69016bf073cb"}

 {
"host": "coder.company.com", 
"path": "/derp", 
"proto": "HTTP/1.1", 
"remote_addr": "172.20.0.4:54062", 
"took": "69.837µs", 
"status_code": 426, 
"latency_ms": 0, 
"response_body": "DERP requires connection upgrade\n", 
"request_id": "143c2491-55dc-4e83-8d69-69016bf073cb"}

 {
"host": "coder.company.com", 
"path": "/derp", 
"proto": "HTTP/1.1", 
"remote_addr": "172.20.0.4:54062", 
"took": "69.837µs", 
"status_code": 426, 
"latency_ms": 0, 
"response_body": "DERP requires connection upgrade\n", 
"request_id": "143c2491-55dc-4e83-8d69-69016bf073cb"}

 {
"host": "coder.company.com", 
"path": "/derp", 
"proto": "HTTP/1.1", 
"remote_addr": "172.20.0.4:54062", 
"took": "69.837µs", 
"status_code": 426, 
"latency_ms": 0, 
"response_body": "DERP requires connection upgrade\n", 
"request_id": "143c2491-55dc-4e83-8d69-69016bf073cb"}

We're using Apache 2.4 as reverse proxy, with the following configuration for our virtual host:

MDomain coder.company.com
<VirtualHost *:443>

        Define SITE coder
        ServerName ${SITE}.company.com
        Protocols h2 http/1.1
        Define BACKEND http://coder:7080/
        SSLEngine on

        ProxyRequests           Off
        ProxyPreserveHost       On
        ProxyPass / ${BACKEND}
        ProxyPassReverse / ${BACKEND}
        
        RewriteEngine On
        RewriteCond %{HTTP:Upgrade} =websocket [NC]
        RewriteRule /(.*) ws://coder:7080/$1 [P,L]
        RewriteCond %{HTTP:Upgrade} !=websocket [NC]
        RewriteRule /(.*) http://coder:7080/$1 [P,L]

</VirtualHost>

MDomain coder.company.com
<VirtualHost *:443>

        Define SITE coder
        ServerName ${SITE}.company.com
        Protocols h2 http/1.1
        Define BACKEND http://coder:7080/
        SSLEngine on

        ProxyRequests           Off
        ProxyPreserveHost       On
        ProxyPass / ${BACKEND}
        ProxyPassReverse / ${BACKEND}
        
        RewriteEngine On
        RewriteCond %{HTTP:Upgrade} =websocket [NC]
        RewriteRule /(.*) ws://coder:7080/$1 [P,L]
        RewriteCond %{HTTP:Upgrade} !=websocket [NC]
        RewriteRule /(.*) http://coder:7080/$1 [P,L]

</VirtualHost>

MDomain coder.company.com
<VirtualHost *:443>

        Define SITE coder
        ServerName ${SITE}.company.com
        Protocols h2 http/1.1
        Define BACKEND http://coder:7080/
        SSLEngine on

        ProxyRequests           Off
        ProxyPreserveHost       On
        ProxyPass / ${BACKEND}
        ProxyPassReverse / ${BACKEND}
        
        RewriteEngine On
        RewriteCond %{HTTP:Upgrade} =websocket [NC]
        RewriteRule /(.*) ws://coder:7080/$1 [P,L]
        RewriteCond %{HTTP:Upgrade} !=websocket [NC]
        RewriteRule /(.*) http://coder:7080/$1 [P,L]

</VirtualHost>

MDomain coder.company.com
<VirtualHost *:443>

        Define SITE coder
        ServerName ${SITE}.company.com
        Protocols h2 http/1.1
        Define BACKEND http://coder:7080/
        SSLEngine on

        ProxyRequests           Off
        ProxyPreserveHost       On
        ProxyPass / ${BACKEND}
        ProxyPassReverse / ${BACKEND}
        
        RewriteEngine On
        RewriteCond %{HTTP:Upgrade} =websocket [NC]
        RewriteRule /(.*) ws://coder:7080/$1 [P,L]
        RewriteCond %{HTTP:Upgrade} !=websocket [NC]
        RewriteRule /(.*) http://coder:7080/$1 [P,L]

</VirtualHost>

Not matter what I change, the connection to a workspace is immediately terminated or it hangs until it closes.
With the settings above, it'll hang (not sure how long), and then it'll disconnect.

I must admit, I'm really not all that familiar with Apache configuration, so I apologize if a fix is obvious.

Any ideas?

kyle•9/28/22, 7:16 PM

Try enabling KeepAliveKeepAlive. See: https://ubiq.co/tech-blog/enable-keep-alive-apache/

Ubiq BIUbiq

How to Enable Keep Alive in Apache - Ubiq BI

Keep Alive Header improves website performance and speed. Here's how to enable keep alive in Apache.

Kkyle Try enabling `KeepAlive`. See: https://ubiq.co/tech-blog/enable-keep-alive-apach...

cotheerOP•9/28/22, 7:31 PM

Nothing. The same message keeps showing in the coder server logs (remote_addr has different port each time) and then the server closes the connection.

kyle•9/28/22, 7:32 PM

So the "Upgrade" header is set to "derp" for that route, so I believe the rewrite condition is hurting that somehow.

kyle•9/28/22, 7:33 PM

I believe you'll have to do similarly for the "derp" upgrade type that you did for WebSockets

kyle•9/28/22, 7:34 PM

        RewriteEngine On
        RewriteCond %{HTTP:Upgrade} =websocket [NC]
        RewriteRule /(.*) ws://coder:7080/$1 [P,L]
        RewriteCond %{HTTP:Upgrade} =derp [NC]
        RewriteRule /(.*) http://coder:7080/$1 [P,L]
        RewriteCond %{HTTP:Upgrade} !=websocket [NC]
        RewriteCond %{HTTP:Upgrade} !=derp [NC]
        RewriteRule /(.*) http://coder:7080/$1 [P,L]

        RewriteEngine On
        RewriteCond %{HTTP:Upgrade} =websocket [NC]
        RewriteRule /(.*) ws://coder:7080/$1 [P,L]
        RewriteCond %{HTTP:Upgrade} =derp [NC]
        RewriteRule /(.*) http://coder:7080/$1 [P,L]
        RewriteCond %{HTTP:Upgrade} !=websocket [NC]
        RewriteCond %{HTTP:Upgrade} !=derp [NC]
        RewriteRule /(.*) http://coder:7080/$1 [P,L]

kyle•9/28/22, 7:34 PM

Give that a shot... I believe it should work

Kkyle ``` RewriteEngine On RewriteCond %{HTTP:Upgrade} =websocket [NC]...

cotheerOP•9/28/22, 7:39 PM

No go.

kyle•9/28/22, 7:40 PM

Humph, any new logs? Or the same thing?

cotheerOP•9/28/22, 7:40 PM

Same general message. Here's bit more from the log:

kyle•9/28/22, 7:40 PM

It should actually work with just the WebSocket rule, so I'm not sure why it's failing.

kyle•9/28/22, 7:41 PM

Let's try this instead:

RewriteCond %{HTTP:Connection} Upgrade [NC]
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteRule /(.*) ws://coder:7080/$1 [P,L]
RewriteCond %{HTTP:Connection} Upgrade [NC]
RewriteCond %{HTTP:Upgrade} derp [NC]
RewriteRule /(.*) http://coder:7080/$1 [P,L]

RewriteCond %{HTTP:Connection} Upgrade [NC]
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteRule /(.*) ws://coder:7080/$1 [P,L]
RewriteCond %{HTTP:Connection} Upgrade [NC]
RewriteCond %{HTTP:Upgrade} derp [NC]
RewriteRule /(.*) http://coder:7080/$1 [P,L]

kyle•9/28/22, 7:42 PM

This explicitly separates DERP and WebSockets

cotheerOP•9/28/22, 7:47 PM

2022-09-28 19:45:02.810 [WARN] (coderd) <./coderd/httpmw/logger.go:53> Logger.func1.1 GET {"host": "coder.company.com", "path": "/derp", "proto": "HTTP/1.1", "remote_addr": "172.20.0.4:56978", "took": "75.791µs", "status_code": 426, "latency_ms": 0, "response_body": "DERP requires connection upgrade\n", "request_id": "037e7066-8a95-4838-8c88-c642da45d5a1"}
2022-09-28 19:45:03.062 [DEBUG] (coderd.tailnet.wgengine) <./tailnet/conn.go:567> Logger.func1 [v1] magicsock: disco: timeout waiting for pong 788a1834d246 from 172.17.0.2:43228 ([J05V3], d:a1ef2b366e4870c4)
2022-09-28 19:45:03.062 [DEBUG] (coderd.tailnet.wgengine) <./tailnet/conn.go:567> Logger.func1 [v1] magicsock: disco: timeout waiting for pong f22fe82ba192 from 3.X.X.X:43228 ([J05V3], d:a1ef2b366e4870c4)
2022-09-28 19:45:03.203 [DEBUG] (coderd.tailnet.wgengine) <./tailnet/conn.go:567> Logger.func1 [v2] [J05V3] - Handshake did not complete after 5 seconds, retrying (try 5)
2022-09-28 19:45:03.203 [DEBUG] (coderd.tailnet.wgengine) <./tailnet/conn.go:567> Logger.func1 [v2] [J05V3] - Sending handshake initiation
2022-09-28 19:45:03.204 [DEBUG] (coderd.tailnet.wgengine) <./tailnet/conn.go:567> Logger.func1 [v1] magicsock: disco: send, starting discovery for [J05V3] (d:a1ef2b366e4870c4)
2022-09-28 19:45:03.204 [DEBUG] (coderd.tailnet.wgengine) <./tailnet/conn.go:567> Logger.func1 derphttp.Client.Send: connecting to derp-999 (coder)
2022-09-28 19:45:03.204 [DEBUG] (coderd.tailnet.wgengine) <./tailnet/conn.go:567> Logger.func1 [v1] magicsock: disco: d:41cbdbfdb0c71160->d:a1ef2b366e4870c4 ([J05V3], derp-999) sent call-me-maybe
2022-09-28 19:45:03.205 [DEBUG] (coderd.tailnet.wgengine) <./tailnet/conn.go:567> Logger.func1 [v1] magicsock: disco: d:41cbdbfdb0c71160->d:a1ef2b366e4870c4 ([J05V3], 3.X.X.X:43228) sent ping tx=bdeadf26c4ec

cotheerOP•9/28/22, 7:48 PM

There's the extended log, so same thing with the latest changes.

kyle•9/28/22, 8:33 PM

Interesting. It's odd that it can't upgrade... the code that returns that message only does when up != "websocket" && up != "derp"up != "websocket" && up != "derp"

kyle•9/28/22, 8:33 PM

upup is the value of the UpgradeUpgrade header in this case

Kkyle Interesting. It's odd that it can't upgrade... the code that returns that messag...

cotheerOP•9/28/22, 8:42 PM

Yeah, I've seen the source, I don't get either. Apache modules are loaded and working... and I've spent way too much time on this lol.
For what is worth, I'm using the v0.8.15 docker image.

kyle•9/28/22, 9:25 PM

Try uppercasing DERP maybe? I believe that's how the header is set.

kyle•9/28/22, 9:26 PM

Not sure if the rewrite cond is case-sensitive

cotheerOP•9/29/22, 7:49 AM

For anyone with the same(ish) problem, this is the configuration that worked:

MDomain coder.company.com
<VirtualHost *:443>
        Define SITE coder
        ServerName ${SITE}.company.com
        Protocols h2 http/1.1
        Define BACKEND http://coder:7080/
        SSLEngine on

        #SSLCertificateFile ${SSL_CERT_FILE}
        #SSLCertificateKeyFile ${SSL_CERT_KEYFILE}
        ProxyRequests           Off
        ProxyPreserveHost       On
        ProxyPass / ${BACKEND}
        ProxyPassReverse / ${BACKEND}
        
    # Everything below 
        <Location "/derp">
                RequestHeader set Connection "Upgrade"
                RequestHeader set Upgrade "websocket"
                ProxyPass ws://coder:7080/derp
        </Location>
        
        RewriteEngine On

        RewriteCond %{HTTP:Connection} Upgrade [NC]
        RewriteCond %{HTTP:Upgrade} websocket [NC]
        RewriteRule /(.*) ws://coder:7080/$1 [P,L]

</VirtualHost>

MDomain coder.company.com
<VirtualHost *:443>
        Define SITE coder
        ServerName ${SITE}.company.com
        Protocols h2 http/1.1
        Define BACKEND http://coder:7080/
        SSLEngine on

        #SSLCertificateFile ${SSL_CERT_FILE}
        #SSLCertificateKeyFile ${SSL_CERT_KEYFILE}
        ProxyRequests           Off
        ProxyPreserveHost       On
        ProxyPass / ${BACKEND}
        ProxyPassReverse / ${BACKEND}
        
    # Everything below 
        <Location "/derp">
                RequestHeader set Connection "Upgrade"
                RequestHeader set Upgrade "websocket"
                ProxyPass ws://coder:7080/derp
        </Location>
        
        RewriteEngine On

        RewriteCond %{HTTP:Connection} Upgrade [NC]
        RewriteCond %{HTTP:Upgrade} websocket [NC]
        RewriteRule /(.*) ws://coder:7080/$1 [P,L]

</VirtualHost>

kyle•9/29/22, 2:44 PM

This is awesome! I appreciate you following-up

gabileibo•1/2/23, 3:52 PM

hello @cotheer @kyle !

I had the same issue as you and i copied over your config exactly but i seem to be having another issue

when i just run the proxyy as is i get "DERP requires connection upgrade"

When i copy your config exactly i get "Unexpected Upgrade: DERP (expecting websocket)"

Any ideas on what might be happening?

gabileibo•1/2/23, 5:41 PM

So i ditched apache and went with caddy like the repo example and its all good now

weird situation with the /derp route, it caused alot of headache