Immich•13mo ago

What is my Immich doing now?

It's been processing lots of photos and video overnight.

At this moment I believe something is weird going on behind the scene. Facial Recognition count stays unmoved. No other jobs are running.

Server is non-responsive. CPU cranks to 100%.

Ctrl-C has no effect.

I want to see if I can get some meaningful log and inspect what happened.

Please suggest some methods.

Screen_Shot_2024-12-24_at_2.11.52_PM.png

Immich•12/24/24, 6:15 AM

:wave: Hey @BitePa,

Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.

References

Container Logs:
```
docker compose logs
```
```
docker compose logs
```
docs
Container Status:
```
docker ps -a
```
```
docker ps -a
```
docs
Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA

Immich•12/24/24, 6:15 AM

Checklist

I have...

:blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
:blue_square: read applicable release notes.
:blue_square: reviewed the FAQs for known issues.
:blue_square: reviewed Github for known issues.
:blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
:blue_square: uploaded the relevant information (see below).
:blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

Information

In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:

Your docker-compose.yml and .env files.
Logs from all the containers and their status (see above).
All the troubleshooting steps you've tried so far.
Any recent changes you've made to Immich or your system.
Details about your system (both software/OS and hardware).
Details about your storage (filesystems, type of disks, output of commands like
```
fdisk -l
```
```
fdisk -l
```
and
```
df -h
```
```
df -h
```
).
The version of the Immich server, mobile app, and other relevant pieces.
Any other information that you think might be relevant.

Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)

If this ticket can be closed you can use the

/close

/close

command, and re-open it later if needed.

BitePaOP•12/24/24, 6:21 AM

The Immich server is still pounding the External Libraries server.

Zhigang•12/24/24, 7:10 AM

I think you may need to check facial recognition settings, whether the path are correct. I had met once due to I put the facial recognition model to the wrong path. After moved to correct path, rebuild, run again and then the counts moving.

BitePaOP•12/24/24, 8:18 AM

This is the 2nd round of batch processing. So if this batch and previous batch are put in the same path, the. There’s something wrong. I’ve not changed the path.

BitePaOP•12/24/24, 8:20 AM

Ah... finally it caught my Ctrl-C.

BitePaOP•12/24/24, 8:21 AM

it has this:

BitePaOP•12/24/24, 8:21 AM

immich_redis | 1:M 24 Dec 2024 06:03:04.194 * Background saving started by pid 13342
immich_machine_learning | [12/24/24 14:27:54] CRITICAL WORKER TIMEOUT (pid:13306)
immich_redis | 13342:C 24 Dec 2024 06:27:25.513 * DB saved on disk

BitePaOP•12/24/24, 8:21 AM

How should I proceed and get the log without starting Immich?

BitePaOP•12/25/24, 3:19 AM

Anyone please?

Mraedis•12/25/24, 11:13 AM

Any hardware details? Compose env files?

BitePaOP•12/25/24, 3:48 PM

I’m using Proxmox LXC container. Assigned tons of RAM n CPUs to it. The first batch ran fine, although number of files less than the 2nd batch.

BitePaOP•12/26/24, 3:50 AM

docker-compose.yaml is as follow:

BitePaOP•12/26/24, 3:50 AM

all standard stuff, except the 2 lines for External Libraries

BitePaOP•12/26/24, 3:50 AM

message.txt3.04KB

BitePaOP•12/26/24, 7:29 AM

Ok. I can't wait, so I restarted the docker container. Here is the error when it crashed the whole server:

BitePaOP•12/26/24, 7:29 AM

anything that I can collect for this crash?

message.txt4.82KB

BitePaOP•12/26/24, 7:30 AM

There are lots of "connection timeout".

Mraedis•12/26/24, 7:40 AM

compose seems OK, are the library mounts on the network or on the device?

Mraedis•12/26/24, 7:40 AM

Also are you still scanning every 20 minutes?

BitePaOP•12/26/24, 7:42 AM

I guess I need to dig into postgres or redis.

MMraedis Also are you still scanning every 20 minutes?

BitePaOP•12/26/24, 7:43 AM

no more 20 mins... every 6 hrs or something like that

MMraedis compose seems OK, are the library mounts on the network or on the device?

BitePaOP•12/26/24, 7:43 AM

network

BitePaOP•12/26/24, 7:43 AM

remote server is intact.

Mraedis•12/26/24, 7:44 AM

Just wondering whether the remote device is terminating or the container

BitePaOP•12/26/24, 7:44 AM

in machine-learning log, I got this:

BitePaOP•12/26/24, 7:44 AM

[2;36m[12/24/24 13:14:12][0m[2;36m [0m[34mINFO [0m Application startup complete.
[2;36m[12/24/24 14:27:54][0m[2;36m [0m[1;7;31mCRITICAL[0m WORKER TIMEOUT [1m([0mpi[1;92md:1330[0m6[1m)[0m
[2;36m[12/24/24 15:12:12][0m[2;36m [0m[1;31mERROR [0m Worker [1m([0mpi[1;92md:1330[0m6[1m)[0m was sent SIGKILL! Perhaps out of
[2;36m [0m memory?

BitePaOP•12/26/24, 7:45 AM

no error in postgres log

BitePaOP•12/26/24, 7:47 AM

redis log:

BitePaOP•12/26/24, 7:47 AM

13342:C 24 Dec 2024 06:27:25.513 * DB saved on disk
13342:C 24 Dec 2024 06:27:46.931 * RDB: 1 MB of memory used by copy-on-write
1:M 24 Dec 2024 06:27:51.902 * Background saving terminated with success
1:C 26 Dec 2024 07:04:18.614 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

BitePaOP•12/26/24, 7:47 AM

is it likely that the container died?

MMraedis Just wondering whether the remote device is terminating or the container

BitePaOP•12/26/24, 7:47 AM

i'll check remote server

Mraedis•12/26/24, 7:48 AM

So there is remote ML?

MMraedis So there is remote ML?

BitePaOP•12/26/24, 7:48 AM

nope. it is not configured. there is a container for ML, so I got the log

Mraedis•12/26/24, 7:49 AM

Ah, just

docker compose logs

docker compose logs

next time, it gets logs for all containers

BitePaOP•12/26/24, 7:49 AM

BitePaOP•12/26/24, 7:54 AM

the remote server hosting the External Libraries is mounted using sshfs

BitePaOP•12/26/24, 7:54 AM

nothing weird in the remote server logs

Mraedis•12/26/24, 8:06 AM

sshfs

Mraedis•12/26/24, 8:11 AM

I'm looking into sshfs a bit and that seems remarkably unstable when "layered', such as used within another application rather than just on the CLI

Mraedis•12/26/24, 8:11 AM

i.e. through docker

MMraedis I'm looking into sshfs a bit and that seems remarkably unstable when "layered', ...

BitePaOP•12/26/24, 8:12 AM

oh....

BitePaOP•12/26/24, 8:12 AM

samba is not stable. cifs not stable. what do you suggest?

Mraedis•12/26/24, 8:12 AM

If they are all not stable maybe it's not the filesystems...

BitePaOP•12/26/24, 8:14 AM

ummm...

BitePaOP•12/26/24, 8:15 AM

i only run Debian 11 or 12.

BitePaOP•12/26/24, 8:15 AM

not even Ubuntu

Mraedis•12/26/24, 8:15 AM

Sure, but your hardware or network might be messing things up

Mraedis•12/26/24, 8:15 AM

I think most people use SMB mounts here

BitePaOP•12/26/24, 8:16 AM

may be it stresses everything to the max during the first time scan

What is my Immich doing now?

References

Checklist

Information

Similar Threads

Similar Threads

References

Checklist

Information

Similar Threads