Spined Posts - Answer Overflow

Spined

•Created by Spined on 9/6/2023 in #help-desk-support

Immich broken after storage went full

I am running immich on my proxmox within a ubuntu CT and I use a 1 TB NVME in ZFS RAID1 as storage. My problem was that I was using ZFS and I allocated Immich way more storage than available on the ZFS Raid. But since it is ZFS it lets me overallocate. I forgot about it and it ran full. Immich didn't know about it since the "os" thought there was still enough storage left. Now my immich is broken and the immich_typesense and immich_server containers do not want to start anymore. typesense container logs:

E20230905 20:12:44.675185   358 node.cpp:497] node default_group:172.18.0.3:8107:8108 init_log_storage failed
E20230905 20:12:44.675192   358 raft_server.cpp:121] Fail to init peering node
E20230905 20:12:44.675666   358 typesense_server_utils.cpp:302] Failed to start peering state
E20230905 20:13:36.750438   357 log.cpp:193] Found corrupted header at offset=4273828, header={term=0, type=0, data_len=0, checksum_type=0, data_checksum=0}, path: /data/state/log

E20230905 20:12:44.675185   358 node.cpp:497] node default_group:172.18.0.3:8107:8108 init_log_storage failed
E20230905 20:12:44.675192   358 raft_server.cpp:121] Fail to init peering node
E20230905 20:12:44.675666   358 typesense_server_utils.cpp:302] Failed to start peering state
E20230905 20:13:36.750438   357 log.cpp:193] Found corrupted header at offset=4273828, header={term=0, type=0, data_len=0, checksum_type=0, data_checksum=0}, path: /data/state/log

immich server container logs

Node.js v18.17.0
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [NestFactory] Starting Nest application...
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +22ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +3ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +99ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] InfraModule dependencies initialized +3ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] AppModule dependencies initialized +1ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] DomainModule dependencies initialized +4ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [SearchService] Running bootstrap
Request #1693945393649: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND typesense"
Request #1693945393649: Sleeping for 4s and then retrying request...
Request #1693945393649: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND typesense"
Request #1693945393649: Sleeping for 4s and then retrying request...

Node.js v18.17.0
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [NestFactory] Starting Nest application...
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +22ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +3ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +99ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] InfraModule dependencies initialized +3ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] AppModule dependencies initialized +1ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] DomainModule dependencies initialized +4ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [SearchService] Running bootstrap
Request #1693945393649: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND typesense"
Request #1693945393649: Sleeping for 4s and then retrying request...
Request #1693945393649: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND typesense"
Request #1693945393649: Sleeping for 4s and then retrying request...

When accessing the immich webserver i get an Error 500. Is there a way to fix this without loosing any data? I already asked on reddit: https://www.reddit.com/r/immich/comments/16az20w/storage_went_full_and_immich_broke/

10 replies

Gaming

Programming