Traefik doesn't allow large size upload

alittlebee · 2023-07-21T09:13:43.432Z

Following these messages https://discord.com/channels/979116623879368755/994044917355663450/1131786157361999972 https://discord.com/channels/979116623879368755/994044917355663450/1131862468034035764 I was not able to upload large files via CLI Bulk upload. The file I have isolated is a `MOV` file 2.5Gb in size. When I tried with `immich-proxy` port as a temporary solution, it works, but not when I use Traefik. Since this is a reverse proxy problem, my phone likely suffers from the same issue.

A

alittlebeeOP•7/21/23, 9:14 AM

I've tried this but it did not help.
https://discord.com/channels/979116623879368755/1131862468034035764/1131866743384571994

E

etnoy•7/21/23, 9:41 AM

Does traefik cache requests on disk? What is the free disk space of the traefik system?

A

Allram•7/21/23, 12:05 PM

Do you happen to use Cloudflare?

Eetnoy Does traefik cache requests on disk? What is the free disk space of the traefik ...

A

Allram•7/21/23, 12:06 PM

Traefik does not cache as default to disk. Only on Enterprise version or by plugins

E

etnoy•7/21/23, 12:44 PM

I'm not familiar with traefik, but I had that issue with nginx.

A

Allram•7/21/23, 7:35 PM

This is my Immich Traefik settings:

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${LIBRARY_LOCATION}:/usr/src/app/upload/library
      - ${UPLOAD_LOCATION}:/usr/src/app/upload/upload
      - ${THUMBS_LOCATION}:/usr/src/app/upload/thumbs
      - ${PROFILE_LOCATION}:/usr/src/app/upload/profile
      - ${VIDEO_LOCATION}:/usr/src/app/upload/encoded-video
      - /tmp/Postgresql/postgresql/:/var/run/postgresql
      - /tmp/Redis/Redis/:/var/run/redis
      - ${IMPORT_LOCATION}:/import
    env_file:
      - .env
    depends_on:
      - typesense
    restart: always
    networks:
      immich:
        ipv4_address: 192.168.30.17
    labels:
    - "TZ=Europe/Oslo"
    - "traefik.enable=true"
    - "traefik.http.routers.immich-api.entryPoints=https"
    - "traefik.http.services.immich-api.loadbalancer.server.port=3001"
    - "traefik.http.routers.immich-api.rule=Host(`photos.domain.org`) && Pathprefix(`/api`)"
    - "traefik.http.middlewares.service-immich-api-strip.stripprefix.prefixes=/api"
    - "traefik.http.routers.immich-api.middlewares=middlewares-secure-headers@file,service-immich-api-strip"
    environment:
      - TZ=Europe/Oslo

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${LIBRARY_LOCATION}:/usr/src/app/upload/library
      - ${UPLOAD_LOCATION}:/usr/src/app/upload/upload
      - ${THUMBS_LOCATION}:/usr/src/app/upload/thumbs
      - ${PROFILE_LOCATION}:/usr/src/app/upload/profile
      - ${VIDEO_LOCATION}:/usr/src/app/upload/encoded-video
      - /tmp/Postgresql/postgresql/:/var/run/postgresql
      - /tmp/Redis/Redis/:/var/run/redis
      - ${IMPORT_LOCATION}:/import
    env_file:
      - .env
    depends_on:
      - typesense
    restart: always
    networks:
      immich:
        ipv4_address: 192.168.30.17
    labels:
    - "TZ=Europe/Oslo"
    - "traefik.enable=true"
    - "traefik.http.routers.immich-api.entryPoints=https"
    - "traefik.http.services.immich-api.loadbalancer.server.port=3001"
    - "traefik.http.routers.immich-api.rule=Host(`photos.domain.org`) && Pathprefix(`/api`)"
    - "traefik.http.middlewares.service-immich-api-strip.stripprefix.prefixes=/api"
    - "traefik.http.routers.immich-api.middlewares=middlewares-secure-headers@file,service-immich-api-strip"
    environment:
      - TZ=Europe/Oslo

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${LIBRARY_LOCATION}:/usr/src/app/upload/library
      - ${UPLOAD_LOCATION}:/usr/src/app/upload/upload
      - ${THUMBS_LOCATION}:/usr/src/app/upload/thumbs
      - ${PROFILE_LOCATION}:/usr/src/app/upload/profile
      - ${VIDEO_LOCATION}:/usr/src/app/upload/encoded-video
      - /tmp/Postgresql/postgresql/:/var/run/postgresql
      - /tmp/Redis/Redis/:/var/run/redis
      - ${IMPORT_LOCATION}:/import
    env_file:
      - .env
    depends_on:
      - typesense
    restart: always
    networks:
      immich:
        ipv4_address: 192.168.30.17
    labels:
    - "TZ=Europe/Oslo"
    - "traefik.enable=true"
    - "traefik.http.routers.immich-api.entryPoints=https"
    - "traefik.http.services.immich-api.loadbalancer.server.port=3001"
    - "traefik.http.routers.immich-api.rule=Host(`photos.domain.org`) && Pathprefix(`/api`)"
    - "traefik.http.middlewares.service-immich-api-strip.stripprefix.prefixes=/api"
    - "traefik.http.routers.immich-api.middlewares=middlewares-secure-headers@file,service-immich-api-strip"
    environment:
      - TZ=Europe/Oslo

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${LIBRARY_LOCATION}:/usr/src/app/upload/library
      - ${UPLOAD_LOCATION}:/usr/src/app/upload/upload
      - ${THUMBS_LOCATION}:/usr/src/app/upload/thumbs
      - ${PROFILE_LOCATION}:/usr/src/app/upload/profile
      - ${VIDEO_LOCATION}:/usr/src/app/upload/encoded-video
      - /tmp/Postgresql/postgresql/:/var/run/postgresql
      - /tmp/Redis/Redis/:/var/run/redis
      - ${IMPORT_LOCATION}:/import
    env_file:
      - .env
    depends_on:
      - typesense
    restart: always
    networks:
      immich:
        ipv4_address: 192.168.30.17
    labels:
    - "TZ=Europe/Oslo"
    - "traefik.enable=true"
    - "traefik.http.routers.immich-api.entryPoints=https"
    - "traefik.http.services.immich-api.loadbalancer.server.port=3001"
    - "traefik.http.routers.immich-api.rule=Host(`photos.domain.org`) && Pathprefix(`/api`)"
    - "traefik.http.middlewares.service-immich-api-strip.stripprefix.prefixes=/api"
    - "traefik.http.routers.immich-api.middlewares=middlewares-secure-headers@file,service-immich-api-strip"
    environment:
      - TZ=Europe/Oslo

A

Allram•7/21/23, 7:35 PM

And Immich-web:

immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always
    networks:
      immich:
        ipv4_address: 192.168.30.16
    labels:
    - "TZ=Europe/Oslo"
    - "traefik.enable=true"
    - "traefik.http.routers.immich-http.entryPoints=https"
    - "traefik.http.services.immich-http.loadbalancer.server.port=3000"
    - "traefik.http.routers.immich-http.rule=Host(`photos.domain.org`)"
    - "traefik.http.routers.immich-http.middlewares=middlewares-secure-headers@file"
    environment:
      - TZ=Europe/Oslo

immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always
    networks:
      immich:
        ipv4_address: 192.168.30.16
    labels:
    - "TZ=Europe/Oslo"
    - "traefik.enable=true"
    - "traefik.http.routers.immich-http.entryPoints=https"
    - "traefik.http.services.immich-http.loadbalancer.server.port=3000"
    - "traefik.http.routers.immich-http.rule=Host(`photos.domain.org`)"
    - "traefik.http.routers.immich-http.middlewares=middlewares-secure-headers@file"
    environment:
      - TZ=Europe/Oslo

immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always
    networks:
      immich:
        ipv4_address: 192.168.30.16
    labels:
    - "TZ=Europe/Oslo"
    - "traefik.enable=true"
    - "traefik.http.routers.immich-http.entryPoints=https"
    - "traefik.http.services.immich-http.loadbalancer.server.port=3000"
    - "traefik.http.routers.immich-http.rule=Host(`photos.domain.org`)"
    - "traefik.http.routers.immich-http.middlewares=middlewares-secure-headers@file"
    environment:
      - TZ=Europe/Oslo

immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always
    networks:
      immich:
        ipv4_address: 192.168.30.16
    labels:
    - "TZ=Europe/Oslo"
    - "traefik.enable=true"
    - "traefik.http.routers.immich-http.entryPoints=https"
    - "traefik.http.services.immich-http.loadbalancer.server.port=3000"
    - "traefik.http.routers.immich-http.rule=Host(`photos.domain.org`)"
    - "traefik.http.routers.immich-http.middlewares=middlewares-secure-headers@file"
    environment:
      - TZ=Europe/Oslo

`

A

Allram•7/21/23, 7:37 PM

I also run:

        - crowdsec-bouncer-plugin@file
        - middlewares-compress@file
        - middlewares-rate-limit@file

        - crowdsec-bouncer-plugin@file
        - middlewares-compress@file
        - middlewares-rate-limit@file

As plugins in Traefik, can you send your traefikk.yml and relevant config from fileconfig.yml @alittlebee ?

A

alittlebeeOP•7/24/23, 1:04 PM

Thanks @Allram , I will test out the configuration and let you know about the result. That looks promising

A

alittlebeeOP•7/26/23, 8:54 AM

@Allram I have a very simple traefik configuration. All configurations are done through this docker-compose file and no

traefik.yml

traefik.yml

traefik_docker-compose.txt4.32KB

A

alittlebeeOP•7/26/23, 8:55 AM

And this is my immich config, which works for smaller files.

immich_docker-compose_.txt4.32KB

A

alittlebeeOP•7/26/23, 8:58 AM

Would you mind to share your

traefik.yml

traefik.yml

config and how to load it from the docker container?

A

alittlebeeOP•7/26/23, 9:10 AM

I updated to this and Traefik throws errors

A

alittlebeeOP•7/26/23, 9:10 AM

A

Allram•7/26/23, 9:12 AM

Try to remove middleware 'middlewares-secure-headers@file'

A

Allram•7/26/23, 9:12 AM

That is in my fileconfig.yml so it might be that yours are named different or not exist

A

Allram•7/26/23, 9:13 AM

I can share my files later when I'm in front of my PC

A

alittlebeeOP•7/26/23, 9:18 AM

Thank you. Removining

middlewares-secure-headers@file

middlewares-secure-headers@file

returns my config to the original state, so it works. Looking forward to see your config

A

Allram•7/27/23, 8:09 AM

@alittlebee
Traefik.yml: https://jpst.it/3jDid
Fileconfig.yml: https://jpst.it/3jDjt

A

alittlebeeOP•7/27/23, 3:05 PM

Thanks, @Allram . Wow you have a hardcore configuration.

A

alittlebeeOP•7/27/23, 3:06 PM

I'll try it out and keep you posted

A

Allram•7/27/23, 3:12 PM

Trying to keep it as secure as I can

A

alittlebeeOP•7/27/23, 9:49 PM

Did you follow this guide to setup?
https://docs.ibracorp.io/traefik/master/docker-compose/config-files-explained

This is the only resource I found mentioning fileConfig.yml

Config Files Explained

A

alittlebeeOP•7/27/23, 10:59 PM

Btw, it is mentioned that

traefik.yml

traefik.yml

is a "static config" and

fileConfig

fileConfig

is "dynamic config". What do they mean?

A

alittlebeeOP•7/27/23, 11:23 PM

Ah, I think I start to understand it now.

A

Allram•7/28/23, 4:41 AM

Yes, followed that and a few other I found around the internet.

The static traefik.yml, Traefik will only incorporate the changes when it's restarted.

The dynamic fileconfig.yml will be read/updated often by Traefik, so changes can be adapted "on-the-fly" without restarting Traefik.

A

alittlebeeOP•7/28/23, 9:07 AM

Thanks a lot for the explaination. Now I understand.

A

alittlebeeOP•7/28/23, 9:36 AM

At this point, I am pretty sure I configured everything correctly but none helped. Immich now runs as normal. I loaded a new middleware

middlewares-immich-secure-headers

middlewares-immich-secure-headers

    middlewares-immich-secure-headers:
      headers:
        accessControlAllowMethods:
          - GET
          - OPTIONS
          - PUT
        accessControlMaxAge: 100
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        # sslRedirect: true #replaced with middlewares-https-redirectscheme for v2.5.x
        stsSeconds: 63072000
        stsIncludeSubdomains: true
        stsPreload: true
        forceSTSHeader: true
        # frameDeny: true #overwritten by customFrameOptionsValue
        customFrameOptionsValue: "SAMEORIGIN" #CSP takes care of this but may be needed for organizr.
        contentTypeNosniff: true
        browserXssFilter: true
        # sslForceHost: true # add sslHost to all of the services
        # sslHost: "{{env "DOMAINNAME_SHB"}}"
        referrerPolicy: "same-origin"
        # Setting contentSecurityPolicy is more secure but it can break things. Proper auth will reduce the risk.
        # the below line also breaks some apps due to 'none' - sonarr, radarr, etc.
        # contentSecurityPolicy: "frame-ancestors '*.{{env "DOMAINNAME_SHB"}}:*';object-src 'none';script-src 'none';"
        # Line below, featurePolicy, was deprecated in v2.5.x in favor permissionPolicy
        # featurePolicy: "camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';"
        permissionsPolicy: "camera=(), microphone=(), geolocation=(), payment=(), usb=(), vr=()"
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex,"
          server: ""
          # https://community.traefik.io/t/how-to-make-websockets-work-with-traefik-2-0-setting-up-rancher/1732
          X-Forwarded-Proto: "https"

    middlewares-immich-secure-headers:
      headers:
        accessControlAllowMethods:
          - GET
          - OPTIONS
          - PUT
        accessControlMaxAge: 100
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        # sslRedirect: true #replaced with middlewares-https-redirectscheme for v2.5.x
        stsSeconds: 63072000
        stsIncludeSubdomains: true
        stsPreload: true
        forceSTSHeader: true
        # frameDeny: true #overwritten by customFrameOptionsValue
        customFrameOptionsValue: "SAMEORIGIN" #CSP takes care of this but may be needed for organizr.
        contentTypeNosniff: true
        browserXssFilter: true
        # sslForceHost: true # add sslHost to all of the services
        # sslHost: "{{env "DOMAINNAME_SHB"}}"
        referrerPolicy: "same-origin"
        # Setting contentSecurityPolicy is more secure but it can break things. Proper auth will reduce the risk.
        # the below line also breaks some apps due to 'none' - sonarr, radarr, etc.
        # contentSecurityPolicy: "frame-ancestors '*.{{env "DOMAINNAME_SHB"}}:*';object-src 'none';script-src 'none';"
        # Line below, featurePolicy, was deprecated in v2.5.x in favor permissionPolicy
        # featurePolicy: "camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';"
        permissionsPolicy: "camera=(), microphone=(), geolocation=(), payment=(), usb=(), vr=()"
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex,"
          server: ""
          # https://community.traefik.io/t/how-to-make-websockets-work-with-traefik-2-0-setting-up-rancher/1732
          X-Forwarded-Proto: "https"

    middlewares-immich-secure-headers:
      headers:
        accessControlAllowMethods:
          - GET
          - OPTIONS
          - PUT
        accessControlMaxAge: 100
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        # sslRedirect: true #replaced with middlewares-https-redirectscheme for v2.5.x
        stsSeconds: 63072000
        stsIncludeSubdomains: true
        stsPreload: true
        forceSTSHeader: true
        # frameDeny: true #overwritten by customFrameOptionsValue
        customFrameOptionsValue: "SAMEORIGIN" #CSP takes care of this but may be needed for organizr.
        contentTypeNosniff: true
        browserXssFilter: true
        # sslForceHost: true # add sslHost to all of the services
        # sslHost: "{{env "DOMAINNAME_SHB"}}"
        referrerPolicy: "same-origin"
        # Setting contentSecurityPolicy is more secure but it can break things. Proper auth will reduce the risk.
        # the below line also breaks some apps due to 'none' - sonarr, radarr, etc.
        # contentSecurityPolicy: "frame-ancestors '*.{{env "DOMAINNAME_SHB"}}:*';object-src 'none';script-src 'none';"
        # Line below, featurePolicy, was deprecated in v2.5.x in favor permissionPolicy
        # featurePolicy: "camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';"
        permissionsPolicy: "camera=(), microphone=(), geolocation=(), payment=(), usb=(), vr=()"
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex,"
          server: ""
          # https://community.traefik.io/t/how-to-make-websockets-work-with-traefik-2-0-setting-up-rancher/1732
          X-Forwarded-Proto: "https"

    middlewares-immich-secure-headers:
      headers:
        accessControlAllowMethods:
          - GET
          - OPTIONS
          - PUT
        accessControlMaxAge: 100
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        # sslRedirect: true #replaced with middlewares-https-redirectscheme for v2.5.x
        stsSeconds: 63072000
        stsIncludeSubdomains: true
        stsPreload: true
        forceSTSHeader: true
        # frameDeny: true #overwritten by customFrameOptionsValue
        customFrameOptionsValue: "SAMEORIGIN" #CSP takes care of this but may be needed for organizr.
        contentTypeNosniff: true
        browserXssFilter: true
        # sslForceHost: true # add sslHost to all of the services
        # sslHost: "{{env "DOMAINNAME_SHB"}}"
        referrerPolicy: "same-origin"
        # Setting contentSecurityPolicy is more secure but it can break things. Proper auth will reduce the risk.
        # the below line also breaks some apps due to 'none' - sonarr, radarr, etc.
        # contentSecurityPolicy: "frame-ancestors '*.{{env "DOMAINNAME_SHB"}}:*';object-src 'none';script-src 'none';"
        # Line below, featurePolicy, was deprecated in v2.5.x in favor permissionPolicy
        # featurePolicy: "camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';"
        permissionsPolicy: "camera=(), microphone=(), geolocation=(), payment=(), usb=(), vr=()"
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex,"
          server: ""
          # https://community.traefik.io/t/how-to-make-websockets-work-with-traefik-2-0-setting-up-rancher/1732
          X-Forwarded-Proto: "https"

with the same content as you.

A

alittlebeeOP•7/28/23, 9:36 AM

The error is still there

A

alittlebeeOP•7/28/23, 11:16 AM

This is my traefik for the

immich_server

immich_server

A

Allram•7/28/23, 11:34 AM

This is weird

and you don't have Cloudflare or anything in front? Might it be the timeouts that need to be increased?

J

jrasm91•7/28/23, 11:51 AM

What is the actual error message?

Jjrasm91 What is the actual error message?

A

alittlebeeOP•7/31/23, 10:52 AM

It is the same as my original error, this one.
Traefik doesn't allow large size upload

AAllram This is weird 😅 and you don't have Cloudflare or anything in front? Might it be...

A

alittlebeeOP•7/31/23, 10:52 AM

I only use CloudFlare to issue certificate/DNS challenge. Not sure if it needs to be configured with something.

A

Allram•7/31/23, 11:19 AM

No, was just thinking if you use Cloudflare's proxy service, since that will limit to 100MB transfers

A

alittlebeeOP•7/31/23, 11:47 AM

Ah, no. I ran

cloudflared

cloudflared

before and ditched it as soon as I discovered the limitation.

Aalittlebee It is the same as my original error, this one. https://discord.com/channels/9791...

J

jrasm91•7/31/23, 12:34 PM

I don't see an error. You just say "unable to upload large video". Did you get an actual error somewhere?

A

alittlebeeOP•7/31/23, 12:35 PM

I linked the error in the message. But here you go.

J

jrasm91•7/31/23, 12:38 PM

Sorry I clicked through a bit but could not locate it. Can you send a screenshot of the error further down? The "response" section might have some more details.

A

alittlebeeOP•7/31/23, 12:47 PM

Of course, here you go

Immich_error_logs5.12KB

J

jrasm91•7/31/23, 2:11 PM

 data: '<html>\r\n' +
        '<head><title>500 Internal Server Error</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>500 Internal Server Error</h1></center>\r\n' +
        '<hr><center>cloudflare</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'

 data: '<html>\r\n' +
        '<head><title>500 Internal Server Error</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>500 Internal Server Error</h1></center>\r\n' +
        '<hr><center>cloudflare</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'

 data: '<html>\r\n' +
        '<head><title>500 Internal Server Error</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>500 Internal Server Error</h1></center>\r\n' +
        '<hr><center>cloudflare</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'

 data: '<html>\r\n' +
        '<head><title>500 Internal Server Error</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>500 Internal Server Error</h1></center>\r\n' +
        '<hr><center>cloudflare</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'

J

jrasm91•7/31/23, 2:11 PM

There is a reference to cloudflare here. Hmm

A

A2251•7/31/23, 2:14 PM

I have the same problem now. Any larger videos do not get uploaded via cloudfared due to size limitation so perhaps some type of chunking can be enabled OR "trusted / local" wifi AP can be added to the list in which the immich instance is routed to a local ip vs external. Similar to how home assistant does the implementation

A

A2251•7/31/23, 2:15 PM

When i say now - this is because i lost an ability to open ports & have to use cloudfared tunnels to expose anything out of my local network due to internet provider change.

AA2251 I have the same problem now. Any larger videos do not get uploaded via cloudfare...

J

jrasm91•7/31/23, 2:18 PM

This gets asked quite a bit. I don't really care either way, but you'd have to convince a mobile dev to add it. cc: @Alex

A

alittlebeeOP•7/31/23, 6:42 PM

I don't use Cloudflared. I use Traefik and Cloudflare is only used to verify DNS

Aalittlebee I don't use Cloudflared. I use Traefik and Cloudflare is only used to verify DNS

J

jrasm91•7/31/23, 6:46 PM

Then how are you getting a 500 error from cloudflare?

Traefik doesn't allow large size upload

Similar Threads

Similar Threads

Similar Threads