Dokploy•2mo ago

Significant HTTPS slow down

Hi, I was hoping someone might be able to help me with an issue I'm experiencing. I have Dokploy installed on a VPS and I have several apps installed. However I was noticing a significant reduction in download throughput. After some investigation I have discovered that SSL / HTTPS seems to be the culprit. If I setup the same application, in this case it was openspeedtest/latest, one with http and the other https via letsencrypt I see a 10x+ difference in download speeds. I have tried combing through the traefik and dokploy docs for answers (and the internet at large) but haven't found anything definitive on how to deal with this issue so I was wondering if any here may have some insight on how to deal with it? For some additional context my dokploy is using standard configuration and the VPS has ARM64 cpu's (Ampere) in case that is an issue. Any help is appreciated, thanks.

19 Replies

donnieOP•2mo ago

This seems exactly like the issue I'm experiencing: https://community.traefik.io/t/very-bad-upload-speed-when-using-traefik-with-tsl-ssl/15082

Traefik Labs Community Forum

VERY bad Upload Speed when using traefik with TSL / SSL

Few days ago i posted this for me keep getting same error everywhere i try, including a brand new cloud server from linode. After spending few days on the problem, i solved it with the following config. version: "3.9" services: traefik: image: traefik:v2.8.0 container_name: traefik command: - --log.level=INFO - --api.insecure=true - -...

Korma•2mo ago

I can see the same behavior....

Henrik•2mo ago

Does your CPU support AES hardware acceleration? Try changing to something else than Elliptic Curve Cryptography (ECC) to see if you get better performance. It is a trade off between security and performance. Traefik is also among the slower reverse proxies out there

donnieOP•2mo ago

Yes, I believe my VPS does support AES-IN, the results of sudo grep -o aes /proc/cpuinfo are:

aes
aes
aes
aes
aes
aes

aes
aes
aes
aes
aes
aes

I will have a look at the ECC as you suggest @Henrik and see if that makes a difference.

Henrik•2mo ago

No idea what you’ll gain from it, but I’m excited to see the results. You’ll have to downgrade from TLS 1.3 and basically exclusively use TLS 1.2.

donnieOP•2mo ago

I tried adding different curvePreferences and cipherSuites to traefik.yml & dynamic/dokploy.yml to no avail. To be honest I'm not exactly sure what and where I should be updating these settings. Here is what I was attempting to add and I tried many variations of this:

tls:
  options:
    default:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
      curvePreferences:
        - CurveP521
        - CurveP384

tls:
  options:
    default:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
      curvePreferences:
        - CurveP521
        - CurveP384

@Henrik if you have any insight on what exactly I should be using for settings and where to place those in the dokploy traefik config files I'm all ears.

Henrik•2mo ago

TLS12 needs to be the maximum, and remove all ECDHE. You're weakening your security by doing it, but may gain some speed

donnieOP•2mo ago

Okay, I've tried to just set the maxVersion to TLS 1.2 but it just seems to break things. Here is my /etc/dokploy/traefik/traefik.yml file with the updated settings (entryPoints > websecure > http > tls), does this look correct?

providers:
  swarm:
    exposedByDefault: false
    watch: true
  docker:
    exposedByDefault: false
    watch: true
    network: dokploy-network
  file:
    directory: /etc/dokploy/traefik/dynamic
    watch: true
entryPoints:
  web:
    address: ':80'
  websecure:
    address: ':443'
    http3:
      advertisedPort: 443
    http:
      tls:
        certResolver: letsencrypt
        options:
          default:
            maxVersion: VersionTLS12
api:
  insecure: true
certificatesResolvers:
  letsencrypt:
    acme:
      email: test@localhost.com
      storage: /etc/dokploy/traefik/dynamic/acme.json
      httpChallenge:
        entryPoint: web

providers:
  swarm:
    exposedByDefault: false
    watch: true
  docker:
    exposedByDefault: false
    watch: true
    network: dokploy-network
  file:
    directory: /etc/dokploy/traefik/dynamic
    watch: true
entryPoints:
  web:
    address: ':80'
  websecure:
    address: ':443'
    http3:
      advertisedPort: 443
    http:
      tls:
        certResolver: letsencrypt
        options:
          default:
            maxVersion: VersionTLS12
api:
  insecure: true
certificatesResolvers:
  letsencrypt:
    acme:
      email: test@localhost.com
      storage: /etc/dokploy/traefik/dynamic/acme.json
      httpChallenge:
        entryPoint: web

Henrik•2mo ago

Here's where there is multiple points to debug. Maybe your client refuses to establish a TLS connection with anything lower than 1.3. It's a fun side quest you've started, but I wouldn't really bother myself.

donnieOP•2mo ago

I hear what you're saying but does it not feel crazy to accept a massive reduction in bandwidth performance (-10x)? This severely impacts the applications and services I want to run on my server. Videos chop when streaming, downloads take forever and websites take a long time to load. I feel like there is something fundamentally wrong here as I'm sure most don't face these bottlenecks otherwise there would be more of an uproar no? I've pretty much run out of ideas, this issue is a blocker for me and my projects so, while I really don't want to, I think I'll have to do the walk of shame back to commercial hosting offerings :/

Henrik•2mo ago

Try to enable compression and see if it helps. In my mind it should make it worse, but I saw a reddit user recommend it

Traefik Compress Documentation - Traefik

Traefik Proxy's HTTP middleware lets you compress responses before sending them to the client. Read the technical documentation.

donnieOP•2mo ago

That doesn't appear to have any effect unfortunately :/ @Henrik do you have a significant slow down on your own Dokploy instances when using https? Is this something you just live with or is it an issue you're not experiencing?

Henrik•2mo ago

I'm able to have video chats using my VPS, so I haven't given it that much thought.

donnieOP•2mo ago

So I just tried the same speedtest with a separate server using Coolify. First with Traefik configured as the proxy and then with Caddy. Both had the exact same results where the https speed was severealy restricted. I don't know what's going on 🤦‍♂️

Henrik•2mo ago

Could it be anitvirus on your client side that tries to do SSL inspection? Or have you verified that it is THE SERVER that struggles with TLS?

donnieOP•2mo ago

Oh...my...god. I think it's something in my home network 🤦‍♂️. I had tried a VPN to check location wasn't a factor but I hadn't attempted any of this outside my current network. I just switched my phone over to mobile carrier netowrk only and everything ran just fine in http and https. fml. @Henrik, I'm so sorry to have wasted your time, I should have caught this earlier.

Henrik•2mo ago

Hahahahahahahaha 🤣 I wouldn't say wasted. We both learned something

donnieOP•2mo ago

Haha I can't wait to drop "cipher suite" and "elliptic curve cartography" into casual conversation tonight and watch the eyes roll back into heads 🙂

Henrik•2mo ago

Maybe save those for the next job interview 😉

Gaming

Programming

Significant HTTPS slow down

Did you find this page helpful?