Using Apache HTTPD as reverse proxy: Unexpected Upgrade DERP error

Hello,

I am using Apache HTTPD as reverse proxy, with the following configuration for our virtual host:

<VirtualHost *:443>
  SSLEngine On

  # Certificat
  ...

   ServerName coder.company.com

  ProxyPreserveHost On
  ProxyRequests off
  ProxyPass / http://coder:7080/
  ProxyPassReverse / http://coder:7080/

  <Location "/derp">
    RequestHeader set Connection "Upgrade"
    RequestHeader set Upgrade "websocket"
    Proxypass ws://coder:7080/derp
  </Location>

  RewriteEngine on

  RewriteCond %{HTTP:Connection} Upgrade [NC]
  RewriteCond %{HTTP:Upgrade} websocket [NC]
  RewriteRule /(.*) ws://coder:7080/$1 [P,L]

  # Custom log file for SSL
  ErrorLog /ssl.error.log
  CustomLog /ssl.access.log combined
</VirtualHost>

<VirtualHost *:80>
  ServerName coder.company.com
  RewriteEngine On
  RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>

<VirtualHost *:443>
  SSLEngine On

  # Certificat
  ...

   ServerName coder.company.com

  ProxyPreserveHost On
  ProxyRequests off
  ProxyPass / http://coder:7080/
  ProxyPassReverse / http://coder:7080/

  <Location "/derp">
    RequestHeader set Connection "Upgrade"
    RequestHeader set Upgrade "websocket"
    Proxypass ws://coder:7080/derp
  </Location>

  RewriteEngine on

  RewriteCond %{HTTP:Connection} Upgrade [NC]
  RewriteCond %{HTTP:Upgrade} websocket [NC]
  RewriteRule /(.*) ws://coder:7080/$1 [P,L]

  # Custom log file for SSL
  ErrorLog /ssl.error.log
  CustomLog /ssl.access.log combined
</VirtualHost>

<VirtualHost *:80>
  ServerName coder.company.com
  RewriteEngine On
  RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>

<VirtualHost *:443>
  SSLEngine On

  # Certificat
  ...

   ServerName coder.company.com

  ProxyPreserveHost On
  ProxyRequests off
  ProxyPass / http://coder:7080/
  ProxyPassReverse / http://coder:7080/

  <Location "/derp">
    RequestHeader set Connection "Upgrade"
    RequestHeader set Upgrade "websocket"
    Proxypass ws://coder:7080/derp
  </Location>

  RewriteEngine on

  RewriteCond %{HTTP:Connection} Upgrade [NC]
  RewriteCond %{HTTP:Upgrade} websocket [NC]
  RewriteRule /(.*) ws://coder:7080/$1 [P,L]

  # Custom log file for SSL
  ErrorLog /ssl.error.log
  CustomLog /ssl.access.log combined
</VirtualHost>

<VirtualHost *:80>
  ServerName coder.company.com
  RewriteEngine On
  RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>

<VirtualHost *:443>
  SSLEngine On

  # Certificat
  ...

   ServerName coder.company.com

  ProxyPreserveHost On
  ProxyRequests off
  ProxyPass / http://coder:7080/
  ProxyPassReverse / http://coder:7080/

  <Location "/derp">
    RequestHeader set Connection "Upgrade"
    RequestHeader set Upgrade "websocket"
    Proxypass ws://coder:7080/derp
  </Location>

  RewriteEngine on

  RewriteCond %{HTTP:Connection} Upgrade [NC]
  RewriteCond %{HTTP:Upgrade} websocket [NC]
  RewriteRule /(.*) ws://coder:7080/$1 [P,L]

  # Custom log file for SSL
  ErrorLog /ssl.error.log
  CustomLog /ssl.access.log combined
</VirtualHost>

<VirtualHost *:80>
  ServerName coder.company.com
  RewriteEngine On
  RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>

Coder dashboard works. I can create templates and workspaces. However, connection to a workspace does not work.

From the Apache HTTPD error logs, I can see this error

[Mon Feb 06 07:47:39.762418 2023] [proxy:error] [pid 9:tid 140558106359552] [client 192.168.164.94:37880] AH00898: Unexpected Upgrade: DERP (expecting websocket) returned by /derp

[Mon Feb 06 07:47:39.762418 2023] [proxy:error] [pid 9:tid 140558106359552] [client 192.168.164.94:37880] AH00898: Unexpected Upgrade: DERP (expecting websocket) returned by /derp

[Mon Feb 06 07:47:39.762418 2023] [proxy:error] [pid 9:tid 140558106359552] [client 192.168.164.94:37880] AH00898: Unexpected Upgrade: DERP (expecting websocket) returned by /derp

[Mon Feb 06 07:47:39.762418 2023] [proxy:error] [pid 9:tid 140558106359552] [client 192.168.164.94:37880] AH00898: Unexpected Upgrade: DERP (expecting websocket) returned by /derp

I've seen this issue DERP requires connection upgrade but the proposed workaround to switch to caddy does not suit me.

Any idea to complete my Apache HTTPD configuration?

thy•2/6/23, 6:46 PM

I'm not too familiar with apache, but /derp/derp is a bit special in that it uses a non-standard Upgrade: derpUpgrade: derp header and it's more like a raw TCP connection than a websocket.

thy•2/6/23, 6:48 PM

If you're unable to figure out a config that works you can always run the derp separate from coder with https://hub.docker.com/r/fredliang/derper or use the official Tailscale derp servers with CODER_DERP_CONFIG_URL=https://controlplane.tailscale.com/derpmap/defaultCODER_DERP_CONFIG_URL=https://controlplane.tailscale.com/derpmap/default and CODER_DERP_SERVER_ENABLE=falseCODER_DERP_SERVER_ENABLE=false

mickaelbaronOP•2/7/23, 5:30 PM

Thx @thy for your response. I am a bit confused, but the solution you propose seems quite complex?

thy•2/7/23, 6:00 PM

using CODER_DERP_CONFIG_URL=https://controlplane.tailscale.com/derpmap/defaultCODER_DERP_CONFIG_URL=https://controlplane.tailscale.com/derpmap/default and CODER_DERP_SERVER_ENABLE=falseCODER_DERP_SERVER_ENABLE=false would definitely be the simplest if you don't mind encrypted traffic going through Tailscale's servers

thy•2/7/23, 6:00 PM

https://github.com/juanfont/headscale/blob/main/docs/reverse-proxy.md#apache does this potentially work?

GitHub

headscale/reverse-proxy.md at main · juanfont/headscale

An open source, self-hosted implementation of the Tailscale control server - headscale/reverse-proxy.md at main · juanfont/headscale

mickaelbaronOP•2/13/23, 8:25 AM

Dear @thy your last proposition is working. upgrade=anyupgrade=any is required as a parameter for ProxyPassProxyPass so that WebSockets traffic whose Upgrade header value is not equal to WebSocket. Thx a lot

For all intents and purposes, here is my Apache configuration

<VirtualHost *:443>
        SSLEngine On
        SSLProxyEngine on

        SSLProxyVerify none
        SSLProxyCheckPeerCN off
        SSLProxyCheckPeerName off
        SSLProxyCheckPeerExpire off

        SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
        SSLHonorCipherOrder On
        SSLCipherSuite "..."
        SSLCompression off

        # HSTS (http://fr.wikipedia.org/wiki/HTTP_Strict_Transport_Security)
        Header unset Strict-Transport-Security
        Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        RequestHeader set X-Forwarded-Proto https"
        RequestHeader set X-Forwarded-Port "443"

        # Certificat
        SSLCertificateFile /ssl/fullchain1.pem
        SSLCertificateKeyFile /ssl/privkey1.pem

        ServerName test.mycompany.fr
        ServerAdmin myemail@company.fr

        ProxyPreserveHost On
        ProxyRequests off
        ProxyPass / http://coder:7080/ upgrade=any
        ProxyPassReverse / http://coder:7080/

        RewriteEngine on

        RewriteCond %{HTTP:Connection} Upgrade [NC]
        RewriteCond %{HTTP:Upgrade} websocket [NC]
        RewriteRule /(.*) ws://coder:7080/$1 [P,L]

        # Custom log file for SSL
        ErrorLog /ssl.error.log
        CustomLog /ssl.access.log combined
</VirtualHost>

<VirtualHost *:80>
        ServerName test.mycompany.fr
        ServerAdmin myemail@company.fr
        RewriteEngine On
        RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>

<VirtualHost *:443>
        SSLEngine On
        SSLProxyEngine on

        SSLProxyVerify none
        SSLProxyCheckPeerCN off
        SSLProxyCheckPeerName off
        SSLProxyCheckPeerExpire off

        SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
        SSLHonorCipherOrder On
        SSLCipherSuite "..."
        SSLCompression off

        # HSTS (http://fr.wikipedia.org/wiki/HTTP_Strict_Transport_Security)
        Header unset Strict-Transport-Security
        Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        RequestHeader set X-Forwarded-Proto https"
        RequestHeader set X-Forwarded-Port "443"

        # Certificat
        SSLCertificateFile /ssl/fullchain1.pem
        SSLCertificateKeyFile /ssl/privkey1.pem

        ServerName test.mycompany.fr
        ServerAdmin myemail@company.fr

        ProxyPreserveHost On
        ProxyRequests off
        ProxyPass / http://coder:7080/ upgrade=any
        ProxyPassReverse / http://coder:7080/

        RewriteEngine on

        RewriteCond %{HTTP:Connection} Upgrade [NC]
        RewriteCond %{HTTP:Upgrade} websocket [NC]
        RewriteRule /(.*) ws://coder:7080/$1 [P,L]

        # Custom log file for SSL
        ErrorLog /ssl.error.log
        CustomLog /ssl.access.log combined
</VirtualHost>

<VirtualHost *:80>
        ServerName test.mycompany.fr
        ServerAdmin myemail@company.fr
        RewriteEngine On
        RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>

matifali•2/20/23, 10:16 AM

Dies this also work with VS COde Desktop and ssh connections?

matifali•2/20/23, 10:17 AM

So only thing to add was? upgrade=anyupgrade=any

mickaelbaronOP•3/8/23, 3:14 PM

@matifali Yeah all features are working

Using Apache HTTPD as reverse proxy: Unexpected Upgrade DERP error

Similar Threads

Similar Threads

Similar Threads