Immich•2y ago

Immich broken after storage went full

I am running immich on my proxmox within a ubuntu CT and I use a 1 TB NVME in ZFS RAID1 as storage. My problem was that I was using ZFS and I allocated Immich way more storage than available on the ZFS Raid. But since it is ZFS it lets me overallocate. I forgot about it and it ran full. Immich didn't know about it since the "os" thought there was still enough storage left. Now my immich is broken and the immich_typesense and immich_server containers do not want to start anymore. typesense container logs:

E20230905 20:12:44.675185   358 node.cpp:497] node default_group:172.18.0.3:8107:8108 init_log_storage failed
E20230905 20:12:44.675192   358 raft_server.cpp:121] Fail to init peering node
E20230905 20:12:44.675666   358 typesense_server_utils.cpp:302] Failed to start peering state
E20230905 20:13:36.750438   357 log.cpp:193] Found corrupted header at offset=4273828, header={term=0, type=0, data_len=0, checksum_type=0, data_checksum=0}, path: /data/state/log

E20230905 20:12:44.675185   358 node.cpp:497] node default_group:172.18.0.3:8107:8108 init_log_storage failed
E20230905 20:12:44.675192   358 raft_server.cpp:121] Fail to init peering node
E20230905 20:12:44.675666   358 typesense_server_utils.cpp:302] Failed to start peering state
E20230905 20:13:36.750438   357 log.cpp:193] Found corrupted header at offset=4273828, header={term=0, type=0, data_len=0, checksum_type=0, data_checksum=0}, path: /data/state/log

immich server container logs

Node.js v18.17.0
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [NestFactory] Starting Nest application...
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +22ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +3ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +99ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] InfraModule dependencies initialized +3ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] AppModule dependencies initialized +1ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] DomainModule dependencies initialized +4ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [SearchService] Running bootstrap
Request #1693945393649: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND typesense"
Request #1693945393649: Sleeping for 4s and then retrying request...
Request #1693945393649: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND typesense"
Request #1693945393649: Sleeping for 4s and then retrying request...

Node.js v18.17.0
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [NestFactory] Starting Nest application...
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +22ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +3ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +99ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] InfraModule dependencies initialized +3ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] AppModule dependencies initialized +1ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [InstanceLoader] DomainModule dependencies initialized +4ms
[Nest] 6  - 09/05/2023, 8:23:13 PM     LOG [SearchService] Running bootstrap
Request #1693945393649: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND typesense"
Request #1693945393649: Sleeping for 4s and then retrying request...
Request #1693945393649: Request to Node 0 failed due to "ENOTFOUND getaddrinfo ENOTFOUND typesense"
Request #1693945393649: Sleeping for 4s and then retrying request...

When accessing the immich webserver i get an Error 500. Is there a way to fix this without loosing any data? I already asked on reddit: https://www.reddit.com/r/immich/comments/16az20w/storage_went_full_and_immich_broke/

From the immich community on Reddit

Explore this post and more from the immich community

6 Replies

bo0tzz•2y ago

Try deleting the typesense data volume. This is safe and should fix it

SpinedOP•2y ago

by using

docker volume rm immich-app_tsdata

docker volume rm immich-app_tsdata

bo0tzz•2y ago

Yes

SpinedOP•2y ago

thanks! i think i broke it. tried removing the volume with the command above. Didnt work as it was in use. Removed the container typesense and did a docker compose up again. Now its giving me the following error when composing up:

immich_proxy             | 2023/09/06 18:39:07 [error] 46#46: *162 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.178.110, server: , request: "GET /api/asset/statistics?isArchived=false HTTP/1.1", upstream: "http://172.18.0.5:3001/asset/statistics?isArchived=false", host: "192.168.178.237:2283", referrer: "http://192.168.178.237:2283/photos"

immich_proxy             | 2023/09/06 18:39:07 [error] 46#46: *162 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.178.110, server: , request: "GET /api/asset/statistics?isArchived=false HTTP/1.1", upstream: "http://172.18.0.5:3001/asset/statistics?isArchived=false", host: "192.168.178.237:2283", referrer: "http://192.168.178.237:2283/photos"

` i guess my data is gone

bo0tzz•2y ago

Have you restarted all the containers?

SpinedOP•2y ago

i stopped the compose and started it again. Now everything seems to work Everything is back up again! Thanks a lot. I'll update the reddit thread now so other people with the same error can fix it themselves. ❤️ awesome project

Gaming

Programming

Immich broken after storage went full

Did you find this page helpful?