Immich•2d ago

Failing to start Docker, connection to Valkey/Redis timed out

I just downloaded the latest release's docker-compose.yml file and tried to start Immich, but it just stays there trying to start while timing out on Redis:

immich_server            | Error: connect ETIMEDOUT
immich_server            |     at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
immich_server            |     at Object.onceWrapper (node:events:632:28)
immich_server            |     at Socket.emit (node:events:518:28)
immich_server            |     at Socket._onTimeout (node:net:609:8)
immich_server            |     at listOnTimeout (node:internal/timers:594:17)
immich_server            |     at process.processTimers (node:internal/timers:529:7) {
immich_server            |   errorno: 'ETIMEDOUT',
immich_server            |   code: 'ETIMEDOUT',
immich_server            |   syscall: 'connect'
immich_server            | }

immich_server            | Error: connect ETIMEDOUT
immich_server            |     at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
immich_server            |     at Object.onceWrapper (node:events:632:28)
immich_server            |     at Socket.emit (node:events:518:28)
immich_server            |     at Socket._onTimeout (node:net:609:8)
immich_server            |     at listOnTimeout (node:internal/timers:594:17)
immich_server            |     at process.processTimers (node:internal/timers:529:7) {
immich_server            |   errorno: 'ETIMEDOUT',
immich_server            |   code: 'ETIMEDOUT',
immich_server            |   syscall: 'connect'
immich_server            | }

This seems to be not new, but nowhere could I find any info, other than replacing Redis with Valkey, which is already being used in the latest release. Am I doing something stupidly, horribly wrong, or is the release simply b0rked? I am on Debian stable with docker-ce and the compose plugin from Docker directly (5:28.1.1-1~debian.12~bookworm). compose.yml and .env file are exactly as available on Github. Complete logs are attached.

message.txt

27 Replies

Immich•2d ago

:wave: Hey @madduck, Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:. References - Container Logs: docker compose logs docs - Container Status: docker ps -a docs - Reverse Proxy: https://immich.app/docs/administration/reverse-proxy - Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA Checklist I have... 1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time). 2. :ballot_box_with_check: read applicable release notes. 3. :ballot_box_with_check: reviewed the FAQs for known issues. 4. :ballot_box_with_check: reviewed Github for known issues. 5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy). 6. :blue_square: uploaded the relevant information (see below). 7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable (an item can be marked as "complete" by reacting with the appropriate number) Information In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider: - Your docker-compose.yml and .env files. - Logs from all the containers and their status (see above). - All the troubleshooting steps you've tried so far. - Any recent changes you've made to Immich or your system. - Details about your system (both software/OS and hardware). - Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h). - The version of the Immich server, mobile app, and other relevant pieces. - Any other information that you think might be relevant. Please paste files and logs with proper code formatting, and especially avoid blurry screenshots. Without the right information we can't work out what the problem is. Help us help you ;) If this ticket can be closed you can use the /close command, and re-open it later if needed.

Tempest•2d ago

Try doing a docker compose down And a Docker compose up

madduckOP•2d ago

I tried that, but no change, @Tempest

Tempest•2d ago

What error are you getting / why isn't it starting? Also, if checking checkbox #6 please confirm that you provide all information that it asks for

madduckOP•2d ago

I've included everything above. It starts all containers, up until:

immich_server            | [Nest] 7  - 05/05/2025, 2:07:34 PM     LOG [Microservices:EventRepository] Initialized websocket server
immich_server            | [Nest] 17  - 05/05/2025, 2:07:35 PM     LOG [Api:EventRepository] Initialized websocket server

immich_server            | [Nest] 7  - 05/05/2025, 2:07:34 PM     LOG [Microservices:EventRepository] Initialized websocket server
immich_server            | [Nest] 17  - 05/05/2025, 2:07:35 PM     LOG [Api:EventRepository] Initialized websocket server

then it sits there for a few seconds, and then the Redis ETIMEDOUT errors start pouring in. I unchecked 5,6,7 since I don't even get as far as the browser, or binding a socket. 😦

Tempest•2d ago

So what's going on with the redis/valkey container? Why isn't it starting? There should be a log. Providing .env and compose file would be beneficial as well

madduckOP•2d ago

When the errors start pouring in, the containers are still "starting":

5203fa741a3f   ghcr.io/immich-app/immich-server:release             "tini -- /bin/bash s…"   3 minutes ago   Up 19 seconds (health: starting)   0.0.0.0:2283->2283/tcp, [::]:2283->2283/tcp   immich_server
3ea3292b13f6   tensorchord/pgvecto-rs:pg14-v0.2.0                   "docker-entrypoint.s…"   3 minutes ago   Up 19 seconds (health: starting)   5432/tcp                                      immich_postgres
488c6e12c7c5   ghcr.io/immich-app/immich-machine-learning:release   "tini -- python -m i…"   3 minutes ago   Up 19 seconds (health: starting)                                                 immich_machine_learning
d444e34a1c99   valkey/valkey:8-bookworm                             "docker-entrypoint.s…"   3 minutes ago   Up 19 seconds (health: starting)   6379/tcp                                      immich_redis

5203fa741a3f   ghcr.io/immich-app/immich-server:release             "tini -- /bin/bash s…"   3 minutes ago   Up 19 seconds (health: starting)   0.0.0.0:2283->2283/tcp, [::]:2283->2283/tcp   immich_server
3ea3292b13f6   tensorchord/pgvecto-rs:pg14-v0.2.0                   "docker-entrypoint.s…"   3 minutes ago   Up 19 seconds (health: starting)   5432/tcp                                      immich_postgres
488c6e12c7c5   ghcr.io/immich-app/immich-machine-learning:release   "tini -- python -m i…"   3 minutes ago   Up 19 seconds (health: starting)                                                 immich_machine_learning
d444e34a1c99   valkey/valkey:8-bookworm                             "docker-entrypoint.s…"   3 minutes ago   Up 19 seconds (health: starting)   6379/tcp                                      immich_redis

About compose and .env file: they are exactly as downloaded. Nothing has changed. Which is why I am so confused. Redis logs are included. Nothing in the logs I can discern to be a problem:

1:M 05 May 2025 14:11:54.699 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 05 May 2025 14:11:54.699 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo
1:M 05 May 2025 14:11:54.699 * Valkey version=8.1.0, bits=64, commit=00000000, modified=0, pid=1, just started
1:M 05 May 2025 14:11:54.699 # Warning: no config file specified, using the default config. In order to specify a config file use valkey-server /path/to/valkey.conf
1:M 05 May 2025 14:11:54.703 * monotonic clock: POSIX clock_gettime
1:M 05 May 2025 14:11:54.705 * Running mode=standalone, port=6379.
1:M 05 May 2025 14:11:54.706 * Server initialized
1:M 05 May 2025 14:11:54.706 * Loading RDB produced by Valkey version 8.1.0
1:M 05 May 2025 14:11:54.706 * RDB age 54 seconds
1:M 05 May 2025 14:11:54.706 * RDB memory usage when created 0.84 Mb
1:M 05 May 2025 14:11:54.706 * Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 05 May 2025 14:11:54.706 * DB loaded from disk: 0.000 seconds
1:M 05 May 2025 14:11:54.706 * Ready to accept connections tcp

1:M 05 May 2025 14:11:54.699 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 05 May 2025 14:11:54.699 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo
1:M 05 May 2025 14:11:54.699 * Valkey version=8.1.0, bits=64, commit=00000000, modified=0, pid=1, just started
1:M 05 May 2025 14:11:54.699 # Warning: no config file specified, using the default config. In order to specify a config file use valkey-server /path/to/valkey.conf
1:M 05 May 2025 14:11:54.703 * monotonic clock: POSIX clock_gettime
1:M 05 May 2025 14:11:54.705 * Running mode=standalone, port=6379.
1:M 05 May 2025 14:11:54.706 * Server initialized
1:M 05 May 2025 14:11:54.706 * Loading RDB produced by Valkey version 8.1.0
1:M 05 May 2025 14:11:54.706 * RDB age 54 seconds
1:M 05 May 2025 14:11:54.706 * RDB memory usage when created 0.84 Mb
1:M 05 May 2025 14:11:54.706 * Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 05 May 2025 14:11:54.706 * DB loaded from disk: 0.000 seconds
1:M 05 May 2025 14:11:54.706 * Ready to accept connections tcp

Tempest•2d ago

How are you running this? What's the host os?

madduckOP•2d ago

Debian stable with Docker CE Docker version 28.1.1, build 4eba377

Tempest•2d ago

In an LXC?

madduckOP•2d ago

No, on a proper VM KVM I am running docker compose up that is it.

Tempest•2d ago

How much ram is available? What's the filesystem?

madduckOP•2d ago

Currently, 8Gb are allocated.

               total        used        free      shared  buff/cache   available
Mem:         8131312      689944     1939068         252     5811828     7441368
Swap:        1851388        1292     1850096

               total        used        free      shared  buff/cache   available
Mem:         8131312      689944     1939068         252     5811828     7441368
Swap:        1851388        1292     1850096

filesystem of the host in general is ext4, but the library is on btrfs

Tempest•2d ago

Which library?

madduckOP•2d ago

UPLOAD_LOCATION=./library

root@gnome:/srv/immich# df . library/ -T
Filesystem                      Type  1K-blocks    Used Available Use% Mounted on
/dev/mapper/gnome--vg-srv       ext4    8803596 6224864   2109948  75% /srv
/dev/mapper/gnome--store-immich btrfs  20971520    5920  20429824   1% /srv/immich/library

root@gnome:/srv/immich# df . library/ -T
Filesystem                      Type  1K-blocks    Used Available Use% Mounted on
/dev/mapper/gnome--vg-srv       ext4    8803596 6224864   2109948  75% /srv
/dev/mapper/gnome--store-immich btrfs  20971520    5920  20429824   1% /srv/immich/library

Tempest•2d ago

Is the database on btrfs?

madduckOP•2d ago

no, the database is on ext4 postgresql that is. redis too or well, I actually don't know where redis is. it's a docker volume, right? so should be on ext4 as well

root@gnome:/srv/docker/volumes/24f1b762c57d694fb2232850c9004b69bb9a533b645cf0d1174c8481a191b7cc/_data# ls -l
total 4
-rw------- 1 systemd-timesync systemd-timesync 89 May  5 16:12 dump.rdb
root@gnome:/srv/docker/volumes/24f1b762c57d694fb2232850c9004b69bb9a533b645cf0d1174c8481a191b7cc/_data# df -T .
Filesystem                Type 1K-blocks    Used Available Use% Mounted on
/dev/mapper/gnome--vg-srv ext4   8803596 6224816   2109996  75% /srv

root@gnome:/srv/docker/volumes/24f1b762c57d694fb2232850c9004b69bb9a533b645cf0d1174c8481a191b7cc/_data# ls -l
total 4
-rw------- 1 systemd-timesync systemd-timesync 89 May  5 16:12 dump.rdb
root@gnome:/srv/docker/volumes/24f1b762c57d694fb2232850c9004b69bb9a533b645cf0d1174c8481a191b7cc/_data# df -T .
Filesystem                Type 1K-blocks    Used Available Use% Mounted on
/dev/mapper/gnome--vg-srv ext4   8803596 6224816   2109996  75% /srv

Tempest•2d ago

I believe so? can you humor me and run systemd-detect-virt and share the output?

madduckOP•2d ago

Output is: kvm

madduckOP•2d ago

I just removed everything and redownloaded everything and same problem, complete typescript attached with logs etc.

message.txt

Tempest•2d ago

did you purge downloaded images beforehand? (unlikely they're an issue, just figured it's worth trying?)

NoMachine•2d ago

can you include

docker ps -a
docker inspect immich_server
docker inspect immich_redis

docker ps -a
docker inspect immich_server
docker inspect immich_redis

madduckOP•2d ago

yeah, I did that yesterday too.

madduckOP•2d ago

Here you go…

message.txt

madduckOP•2d ago

Port 2283 is now actually bound though. But the errors are all over "The connection was reset" when trying to HTTP to that port. After a while, workers are killed, and it starts from the beginning:

immich_server            | microservices worker error: Error: write CONNECT_TIMEOUT database:5432, stack: Error: write CONNECT_TIMEOUT database:5432
immich_server            |     at connectTimedOut (/usr/src/app/node_modules/postgres/cjs/src/connection.js:257:20)
immich_server            |     at Timeout.done [as _onTimeout] (/usr/src/app/node_modules/postgres/cjs/src/connection.js:1035:8)
immich_server            |     at listOnTimeout (node:internal/timers:596:11)
immich_server            |     at process.processTimers (node:internal/timers:529:7)
immich_server            | microservices worker exited with code 1
immich_server            | Killing api process
immich_server exited with code 0
immich_server            | Detected CPU Cores: 4
[…]

immich_server            | microservices worker error: Error: write CONNECT_TIMEOUT database:5432, stack: Error: write CONNECT_TIMEOUT database:5432
immich_server            |     at connectTimedOut (/usr/src/app/node_modules/postgres/cjs/src/connection.js:257:20)
immich_server            |     at Timeout.done [as _onTimeout] (/usr/src/app/node_modules/postgres/cjs/src/connection.js:1035:8)
immich_server            |     at listOnTimeout (node:internal/timers:596:11)
immich_server            |     at process.processTimers (node:internal/timers:529:7)
immich_server            | microservices worker exited with code 1
immich_server            | Killing api process
immich_server exited with code 0
immich_server            | Detected CPU Cores: 4
[…]

so now we have timeouts on redis and on postgres. I suspect that the docker networking setup is broken. I am suspecting a broken nftables integration on Debian. On it… Yeah, the rule was wrong. As soon as I permit traffic on the bridge, it seems to work. Sorry for the noise. This is a Docker problem, not even Debian.

Tempest•2d ago

so docker didn't rewrite your firewall to make it work? wild

madduckOP•2d ago

It presumably does the right thing with iptables, but not (yet) with nftables. I am still researching.

Gaming

Programming

Failing to start Docker, connection to Valkey/Redis timed out

Did you find this page helpful?