Immich broken after storage went full

I am running immich on my proxmox within a ubuntu CT and I use a 1 TB NVME in ZFS RAID1 as storage.
My problem was that I was using ZFS and I allocated Immich way more storage than available on the ZFS Raid. But since it is ZFS it lets me overallocate.
I forgot about it and it ran full. Immich didn't know about it since the "os" thought there was still enough storage left.

Now my immich is broken and the immich_typesense and immich_server containers do not want to start anymore.

typesense container logs:

E20230905 20:12:44.675185   358 node.cpp:497] node default_group:172.18.0.3:8107:8108 init_log_storage failed
E20230905 20:12:44.675192   358 raft_server.cpp:121] Fail to init peering node
E20230905 20:12:44.675666   358 typesense_server_utils.cpp:302] Failed to start peering state
E20230905 20:13:36.750438   357 log.cpp:193] Found corrupted header at offset=4273828, header={term=0, type=0, data_len=0, checksum_type=0, data_checksum=0}, path: /data/state/log

E20230905 20:12:44.675185   358 node.cpp:497] node default_group:172.18.0.3:8107:8108 init_log_storage failed
E20230905 20:12:44.675192   358 raft_server.cpp:121] Fail to init peering node
E20230905 20:12:44.675666   358 typesense_server_utils.cpp:302] Failed to start peering state
E20230905 20:13:36.750438   357 log.cpp:193] Found corrupted header at offset=4273828, header={term=0, type=0, data_len=0, checksum_type=0, data_checksum=0}, path: /data/state/log

immich server container logs

Node.js v18.17.0
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [NestFactory] Starting Nest application...
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +22ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +3ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +99ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] InfraModule dependencies initialized +3ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] AppModule dependencies initialized +1ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] DomainModule dependencies initialized +4ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [SearchService] Running bootstrap
Request #1693945393649: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND typesense"
Request #1693945393649: Sleeping for 4s and then retrying request...
Request #1693945393649: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND typesense"
Request #1693945393649: Sleeping for 4s and then retrying request...

Node.js v18.17.0
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [NestFactory] Starting Nest application...
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +22ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +3ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +99ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] InfraModule dependencies initialized +3ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] AppModule dependencies initialized +1ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] DomainModule dependencies initialized +4ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [SearchService] Running bootstrap
Request #1693945393649: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND typesense"
Request #1693945393649: Sleeping for 4s and then retrying request...
Request #1693945393649: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND typesense"
Request #1693945393649: Sleeping for 4s and then retrying request...

When accessing the immich webserver i get an Error 500.
Is there a way to fix this without loosing any data?
I already asked on reddit: https://www.reddit.com/r/immich/comments/16az20w/storage_went_full_and_immich_broke/

From the immich community on Reddit

Explore this post and more from the immich community

Immich broken after storage went full

E20230905 20:12:44.675185   358 node.cpp:497] node default_group:172.18.0.3:8107:8108 init_log_storage failed
E20230905 20:12:44.675192   358 raft_server.cpp:121] Fail to init peering node
E20230905 20:12:44.675666   358 typesense_server_utils.cpp:302] Failed to start peering state
E20230905 20:13:36.750438   357 log.cpp:193] Found corrupted header at offset=4273828, header={term=0, type=0, data_len=0, checksum_type=0, data_checksum=0}, path: /data/state/log

E20230905 20:12:44.675185   358 node.cpp:497] node default_group:172.18.0.3:8107:8108 init_log_storage failed
E20230905 20:12:44.675192   358 raft_server.cpp:121] Fail to init peering node
E20230905 20:12:44.675666   358 typesense_server_utils.cpp:302] Failed to start peering state
E20230905 20:13:36.750438   357 log.cpp:193] Found corrupted header at offset=4273828, header={term=0, type=0, data_len=0, checksum_type=0, data_checksum=0}, path: /data/state/log

immich server container logs

Node.js v18.17.0
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [NestFactory] Starting Nest application...
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +22ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +3ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +99ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] InfraModule dependencies initialized +3ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] AppModule dependencies initialized +1ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] DomainModule dependencies initialized +4ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [SearchService] Running bootstrap
Request #1693945393649: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND typesense"
Request #1693945393649: Sleeping for 4s and then retrying request...
Request #1693945393649: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND typesense"
Request #1693945393649: Sleeping for 4s and then retrying request...

Node.js v18.17.0
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [NestFactory] Starting Nest application...
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +22ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +3ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +99ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] InfraModule dependencies initialized +3ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] AppModule dependencies initialized +1ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] DomainModule dependencies initialized +4ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [SearchService] Running bootstrap
Request #1693945393649: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND typesense"
Request #1693945393649: Sleeping for 4s and then retrying request...
Request #1693945393649: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND typesense"
Request #1693945393649: Sleeping for 4s and then retrying request...

From the immich community on Reddit

Explore this post and more from the immich community

Immich broken after storage went full

Immich broken after storage went full

Similar Threads

Similar Threads

Similar Threads