Immich•2y ago

Facial recognition double faces, not deleting old ones.

Hello, Today I have opted to re-run the all job on FR for the third time, hoping it will fix my double-face issues I'm having. I have understood that when running the job the faces should be removed when the runs starts, this however is not what is happening for me. I'm afraid that the faces will show up for a third time, further damaging my overview. I have been looking through the logs and I do not find any weird behavior from immich_microservices. I could really use some help with this one as I'm beginning to wonder if my setup is stable currently. Along with this post I have reported a bug on github, where my information is provided. https://github.com/immich-app/immich/issues/2596

GitHub

[BUG] Facial Recognition double faces. · Issue #2596 · immich-app/i...

The bug Facial recognition feature has marked people double in some of my images. See the image below for reference. I can't unfortunately check how this happened, my guess is when I ran the FR...

163 Replies

QuafleyOP•2y ago

It doesn't do this to all images per say, but to some. I'm starting to think that it might be a database issue again, just like when I moved over from the unraid app to docker compose. I have started the job again, I will report back tomorrow morning with the results.

jrasm91•2y ago

I don't think running it again will magically do something different

QuafleyOP•2y ago

Well, the idea was to see if running again would prompt a remove on the faces before starting it did not

jrasm91•2y ago

What version are you on?

QuafleyOP•2y ago

latest, 1.58

jrasm91•2y ago

Something is very strange. The confirmation modal had been released already...

QuafleyOP•2y ago

I know that it still has the old faces because of one picture where I know the cover photo is the same as before.

jrasm91•2y ago

I just checked and the demo env prompts before running all faces btw

QuafleyOP•2y ago

So it works like it should Wonder what is going on for me that it is erroring out like this. or well, not working, no errors. Is there a way for me to check in postgres perhaps? Normally I know things to check, but I'm completely clueless as to how to resolve this one. The only thing I can think of is completely ditching db and creating a new one and uploading all assets again via CLI

QuafleyOP•2y ago

https://i.imgur.com/4lqzdPL.png

Imgur

QuafleyOP•2y ago

hm the database for asset_faces is completely unreadable for human. It seems to remove a majority of the faces, however it also leaves some behind. Did you also check for that jrasm?

jrasm91•2y ago

What is that a picture of?

QuafleyOP•2y ago

at random

jrasm91•2y ago

No I mean the database screenshot. What are the headers for each column?

QuafleyOP•2y ago

QuafleyOP•2y ago

I now have proper access thanks to the suggestion in general to the logs of microservices the only errors I see at the moment are these:

29/05/2023 23:34:32
[Nest] 1  - 05/29/2023, 9:34:32 PM   ERROR [ProcessorService] Unable to run job handler: Error: Warning: Invalid EXIF text encoding for UserComment - /usr/src/app/upload/library/790ea9b9-404b-4233-ae86-48d54db701dc/2017/July/IMG_2291+1.JPG
29/05/2023 23:34:32
[Nest] 1  - 05/29/2023, 9:34:32 PM   ERROR [ProcessorService] Error: Warning: Invalid EXIF text encoding for UserComment - /usr/src/app/upload/library/790ea9b9-404b-4233-ae86-48d54db701dc/2017/July/IMG_2291+1.JPG
29/05/2023 23:34:32
    at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29)
29/05/2023 23:34:32
    at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)
29/05/2023 23:34:32
[Nest] 1  - 05/29/2023, 9:34:32 PM   ERROR [ProcessorService] Object:
29/05/2023 23:34:32
{
29/05/2023 23:34:32
  "id": "ca64c8bd-fb18-4e47-bd53-4b1fa42f2845"
29/05/2023 23:34:32
}
29/05/2023 23:34:32
29/05/2023 23:34:33
    at runNextTicks (node:internal/process/task_queues:60:5)
29/05/2023 23:34:33
    at process.processTimers (node:internal/timers:509:9)
29/05/2023 23:34:33
[Nest] 1  - 05/29/2023, 9:34:33 PM   ERROR [ProcessorService] Object:
29/05/2023 23:34:33
{
29/05/2023 23:34:33
  "id": "6c980d6b-7ea6-4b91-ad2b-9a8762051285"
29/05/2023 23:34:33
}
29/05/2023 23:34:33
29/05/2023 23:34:34
[Nest] 1  - 05/29/2023, 9:34:34 PM    WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=f3450de2-f057-4853-9471-f2ea2c073471)
29/05/2023 23:34:35
[Nest] 1  - 05/29/2023, 9:34:35 PM   ERROR [ProcessorService] Unable to run job handler: Error: Warning: Invalid EXIF text encoding for UserComment - /usr/src/app/upload/library/790ea9b9-404b-4233-ae86-48d54db701dc/2017/August/IMG_2327+1.JPG
29/05/2023 23:34:35
[Nest] 1  - 05/29/2023, 9:34:35 PM   ERROR [ProcessorService] Error: Warning: Invalid EXIF text encoding for UserComment - /usr/src/app/upload/library/790ea9b9-404b-4233-ae86-48d54db701dc/2017/August/IMG_2327+1.JPG
29/05/2023 23:34:35
    at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29)
29/05/2023 23:34:35
    at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)
29/05/2023 23:34:35
[Nest] 1  - 05/29/2023, 9:34:35 PM   ERROR [ProcessorService] Object:
29/05/2023 23:34:35
{
29/05/2023 23:34:35
  "id": "f3450de2-f057-4853-9471-f2ea2c073471"
29/05/2023 23:34:35
}
29/05/2023 23:34:35

29/05/2023 23:34:32
[Nest] 1  - 05/29/2023, 9:34:32 PM   ERROR [ProcessorService] Unable to run job handler: Error: Warning: Invalid EXIF text encoding for UserComment - /usr/src/app/upload/library/790ea9b9-404b-4233-ae86-48d54db701dc/2017/July/IMG_2291+1.JPG
29/05/2023 23:34:32
[Nest] 1  - 05/29/2023, 9:34:32 PM   ERROR [ProcessorService] Error: Warning: Invalid EXIF text encoding for UserComment - /usr/src/app/upload/library/790ea9b9-404b-4233-ae86-48d54db701dc/2017/July/IMG_2291+1.JPG
29/05/2023 23:34:32
    at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29)
29/05/2023 23:34:32
    at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)
29/05/2023 23:34:32
[Nest] 1  - 05/29/2023, 9:34:32 PM   ERROR [ProcessorService] Object:
29/05/2023 23:34:32
{
29/05/2023 23:34:32
  "id": "ca64c8bd-fb18-4e47-bd53-4b1fa42f2845"
29/05/2023 23:34:32
}
29/05/2023 23:34:32
29/05/2023 23:34:33
    at runNextTicks (node:internal/process/task_queues:60:5)
29/05/2023 23:34:33
    at process.processTimers (node:internal/timers:509:9)
29/05/2023 23:34:33
[Nest] 1  - 05/29/2023, 9:34:33 PM   ERROR [ProcessorService] Object:
29/05/2023 23:34:33
{
29/05/2023 23:34:33
  "id": "6c980d6b-7ea6-4b91-ad2b-9a8762051285"
29/05/2023 23:34:33
}
29/05/2023 23:34:33
29/05/2023 23:34:34
[Nest] 1  - 05/29/2023, 9:34:34 PM    WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=f3450de2-f057-4853-9471-f2ea2c073471)
29/05/2023 23:34:35
[Nest] 1  - 05/29/2023, 9:34:35 PM   ERROR [ProcessorService] Unable to run job handler: Error: Warning: Invalid EXIF text encoding for UserComment - /usr/src/app/upload/library/790ea9b9-404b-4233-ae86-48d54db701dc/2017/August/IMG_2327+1.JPG
29/05/2023 23:34:35
[Nest] 1  - 05/29/2023, 9:34:35 PM   ERROR [ProcessorService] Error: Warning: Invalid EXIF text encoding for UserComment - /usr/src/app/upload/library/790ea9b9-404b-4233-ae86-48d54db701dc/2017/August/IMG_2327+1.JPG
29/05/2023 23:34:35
    at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29)
29/05/2023 23:34:35
    at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)
29/05/2023 23:34:35
[Nest] 1  - 05/29/2023, 9:34:35 PM   ERROR [ProcessorService] Object:
29/05/2023 23:34:35
{
29/05/2023 23:34:35
  "id": "f3450de2-f057-4853-9471-f2ea2c073471"
29/05/2023 23:34:35
}
29/05/2023 23:34:35

Immich_machine learning is spamming this over and over:

29/05/2023 23:38:30
INFO:     172.18.0.10:56592 - "POST /object-detection/detect-object HTTP/1.1" 200 OK
29/05/2023 23:38:30
INFO:     172.18.0.10:56656 - "POST /facial-recognition/detect-faces HTTP/1.1" 200 OK
29/05/2023 23:38:30
INFO:     172.18.0.10:56666 - "POST /facial-recognition/detect-faces HTTP/1.1" 200 OK
29/05/2023 23:38:30
INFO:     172.18.0.10:56678 - "POST /sentence-transformer/encode-image HTTP/1.1" 200 OK
29/05/2023 23:38:30
INFO:     172.18.0.10:56694 - "POST /sentence-transformer/encode-image HTTP/1.1" 200 OK
29/05/2023 23:38:31
INFO:     172.18.0.10:56716 - "POST /image-classifier/tag-image HTTP/1.1" 200 OK
29/05/2023 23:38:31
INFO:     172.18.0.10:56730 - "POST /facial-recognition/detect-faces HTTP/1.1" 200 OK
29/05/2023 23:38:31
INFO:     172.18.0.10:56744 - "POST /sentence-transformer/encode-image HTTP/1.1" 200 OK

29/05/2023 23:38:30
INFO:     172.18.0.10:56592 - "POST /object-detection/detect-object HTTP/1.1" 200 OK
29/05/2023 23:38:30
INFO:     172.18.0.10:56656 - "POST /facial-recognition/detect-faces HTTP/1.1" 200 OK
29/05/2023 23:38:30
INFO:     172.18.0.10:56666 - "POST /facial-recognition/detect-faces HTTP/1.1" 200 OK
29/05/2023 23:38:30
INFO:     172.18.0.10:56678 - "POST /sentence-transformer/encode-image HTTP/1.1" 200 OK
29/05/2023 23:38:30
INFO:     172.18.0.10:56694 - "POST /sentence-transformer/encode-image HTTP/1.1" 200 OK
29/05/2023 23:38:31
INFO:     172.18.0.10:56716 - "POST /image-classifier/tag-image HTTP/1.1" 200 OK
29/05/2023 23:38:31
INFO:     172.18.0.10:56730 - "POST /facial-recognition/detect-faces HTTP/1.1" 200 OK
29/05/2023 23:38:31
INFO:     172.18.0.10:56744 - "POST /sentence-transformer/encode-image HTTP/1.1" 200 OK

I'm going to head offline now, I'm going to let the job run and check tomorrow what has happened and if the numbers in DB have gone up as well. Alright, so I have just checked, and FR seems to be identical. I could not find any pictures with triple faces, however the pictures that had double assets before still have them now. There are also assets that have some faces double and some not. So it almost seems like some assets aren't properly done.

QuafleyOP•2y ago

They are also separate. The first image only has 1 image, so probably didn't get deleted.

QuafleyOP•2y ago

The same is the case with the pictures I sent to you @jrasm91 and @Alex. It seems like som3 are left behind by the job, although there is no way of confirming this and it also seems like DB isn't being cleared. What also is odd, is that the image that is presumably stuck, with me and my sister on it, is first in the view of myself and my sister. Ignoring timeline completely. They also have a different ID I will now have a quick look at DB

QuafleyOP•2y ago

Numbers on faces have gone up as well Perhaps I can query the so to say new faces

QuafleyOP•2y ago

I have queried the DB with the following:

SELECT A."personId" 
FROM "asset_faces" A 
INNER JOIN "person" P ON A."personId" = P."id" 
WHERE P."createdAt" > NOW() - INTERVAL '1 day';

SELECT A."personId" 
FROM "asset_faces" A 
INNER JOIN "person" P ON A."personId" = P."id" 
WHERE P."createdAt" > NOW() - INTERVAL '1 day';

These are the new results, as far as I understand:

QuafleyOP•2y ago

I have just tracked down personId, the first one. Unfortunately it just seems like the FR has found a new person in the image, no doubles.

QuafleyOP•2y ago

I did manage to find the assetId of this image, where I might've found some actual duplicates within DB.

message.txt

QuafleyOP•2y ago

https://i.imgur.com/JgAiJ4z.png

Imgur

QuafleyOP•2y ago

If they are this similar we could opt for searching the DB on duplicates?

QuafleyOP•2y ago

I have just now executed the following query:

SELECT COUNT(*) as count, "assetId", "embedding"
FROM "asset_faces"
GROUP BY "assetId", "embedding"
HAVING COUNT(*) > 1;

SELECT COUNT(*) as count, "assetId", "embedding"
FROM "asset_faces"
GROUP BY "assetId", "embedding"
HAVING COUNT(*) > 1;

This has listed all the assetId's, where the embedding is similar. As shown in the image below. As far as I can see there are around 6100 doubles. However, I don't know to what extent my query is accurate. I have checked the assetId which I could find linked to my account, since I can't easily view others libraries. Being 6ab930f2-49e5-483b-9d85-cfb322b212ec, the results are as follows, as shown in double.png. When going in to the UI and searching for the corresponding image I see that there are indeed two faces. With this I have also discovered it isn't an issue related to just my account, but also to all the members on my immich instance (4). I have checked how many times I could see the UUID in my query, and that was five. Below I have displayed the results, where I also see that there are 5 duplicates: https://i.imgur.com/s9wqK9F.png I hope this gives a bit more info, and also steps that perhaps can be taken by you on the test env.

Imgur

QuafleyOP•2y ago

The same AssetId is listed multiple times per double FR embedding. Perhaps it can be achieved by deleting 1 of all with:

WITH cte AS (
  SELECT "assetId", "embedding", row_number() OVER(PARTITION BY "assetId", "embedding" ORDER BY "assetId") AS rn
  FROM "asset_faces"
)
DELETE FROM cte
WHERE rn > 1;

WITH cte AS (
  SELECT "assetId", "embedding", row_number() OVER(PARTITION BY "assetId", "embedding" ORDER BY "assetId") AS rn
  FROM "asset_faces"
)
DELETE FROM cte
WHERE rn > 1;

However, I don't know to what extent this will happen again when I rerun the job or add new images. So I would really like to know if you are able to find duplicates in your tables I personally find this approach extremely risky, but I will create a backup beforehand (duhh) to make sure I have something to fall back on.

jrasm91•2y ago

I assume the person id is different though? This could be explained by the face delete not working when you queue the all job. Can you check the timestamps on the duplicates? Oh I'm just seeing the last screenshot

QuafleyOP•2y ago

Yes it is, since they have been regenerated. However the embedding is exactly the same it seems. The regeneration is an assumption. Yeah I have outlined what steps I have taken And how to reproduce

jrasm91•2y ago

I will have to see if there is ever a reason why face delete would fail

QuafleyOP•2y ago

In the end there are 6k duplicates on a 40k library.

jrasm91•2y ago

You never saw errors in immich-microservices or postgres?

QuafleyOP•2y ago

Not that I could see. What I can do I start the job and closely monitor logs with dozzle? Then we can really see if there are errors. With job I'm referring to FR all But this command can be run on dev

jrasm91•2y ago

Right. That is all I can think of.

QuafleyOP•2y ago

Alright, I will test this later today and report back.

jrasm91•2y ago

I can check if I have duplicates.

QuafleyOP•2y ago

Yes please With that query it becomes apparent right away To add it to this thread as well, I have just updated to 1.59 and ran the FR all job again. It prompted me to it would delete all, which it didn't. No errors in any of the containers. Is there a way to set the logging level?

Alex Tran•2y ago

you can add LOG_LEVEL=verbose to the env file

QuafleyOP•2y ago

Thank you, will test and restart the containers. On job start

30/05/2023 19:39:28
[Nest] 1  - 05/30/2023, 5:39:28 PM   DEBUG [JobService] Handling command: queue=recognize-faces-queue,force=true

30/05/2023 19:39:28
[Nest] 1  - 05/30/2023, 5:39:28 PM   DEBUG [JobService] Handling command: queue=recognize-faces-queue,force=true

within immich_server I also found that if you start, pause, start. The force=true turns in to force=false. Immich_web has reported an issue, however I don't know if I can safely put the logs in here. @Alex It seems, I don't know if this is after 1.59 update, that immich_web is reporting the following error on startup:

30/05/2023 19:38:09
[ERROR] hooks.server.ts [handle]: AxiosError: connect ECONNREFUSED 172.18.0.10:3001
30/05/2023 19:38:09
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16) {
30/05/2023 19:38:09
  port: 3001,
30/05/2023 19:38:09
  address: '172.18.0.10',

30/05/2023 19:38:09
[ERROR] hooks.server.ts [handle]: AxiosError: connect ECONNREFUSED 172.18.0.10:3001
30/05/2023 19:38:09
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16) {
30/05/2023 19:38:09
  port: 3001,
30/05/2023 19:38:09
  address: '172.18.0.10',

Alex Tran•2y ago

Hvae you tried restarting the stack?

QuafleyOP•2y ago

I restarted, and it does not do it again.

QuafleyOP•2y ago

However, recognize faces is stuck on this

QuafleyOP•2y ago

since 1.59 Seemingly no errors in any of the containers

Alex Tran•2y ago

You can remove the redis container, it will remove everything in the queue

QuafleyOP•2y ago

I composed down the whole stack and built it back up again, hit the recognize faces all again. same result This is the usual when starting up the immich_machine-learning

30/05/2023 19:46:44
INFO:     Started server process [1]
30/05/2023 19:46:44
INFO:     Waiting for application startup.
30/05/2023 19:46:45
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
30/05/2023 19:46:45
The `max_size` parameter is deprecated and will be removed in v4.26. Please specify in `size['longest_edge'] instead`.
30/05/2023 19:46:47
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
30/05/2023 19:46:52
/opt/venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:54: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider'
30/05/2023 19:46:52
  warnings.warn(
30/05/2023 19:46:52
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
30/05/2023 19:46:53
model ignore: /cache/models/buffalo_l/1k3d68.onnx landmark_3d_68
30/05/2023 19:46:53
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
30/05/2023 19:46:53
model ignore: /cache/models/buffalo_l/2d106det.onnx landmark_2d_106
30/05/2023 19:46:53
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
30/05/2023 19:46:53
find model: /cache/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0
30/05/2023 19:46:53
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
30/05/2023 19:46:53
model ignore: /cache/models/buffalo_l/genderage.onnx genderage
30/05/2023 19:46:53
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
30/05/2023 19:46:54
find model: /cache/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5
30/05/2023 19:46:54
set det-size: (640, 640)
30/05/2023 19:46:54
INFO:     Application startup complete.
30/05/2023 19:46:54
INFO:     Uvicorn running on http://0.0.0.0:3003 (Press CTRL+C to quit)

30/05/2023 19:46:44
INFO:     Started server process [1]
30/05/2023 19:46:44
INFO:     Waiting for application startup.
30/05/2023 19:46:45
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
30/05/2023 19:46:45
The `max_size` parameter is deprecated and will be removed in v4.26. Please specify in `size['longest_edge'] instead`.
30/05/2023 19:46:47
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
30/05/2023 19:46:52
/opt/venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:54: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider'
30/05/2023 19:46:52
  warnings.warn(
30/05/2023 19:46:52
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
30/05/2023 19:46:53
model ignore: /cache/models/buffalo_l/1k3d68.onnx landmark_3d_68
30/05/2023 19:46:53
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
30/05/2023 19:46:53
model ignore: /cache/models/buffalo_l/2d106det.onnx landmark_2d_106
30/05/2023 19:46:53
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
30/05/2023 19:46:53
find model: /cache/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0
30/05/2023 19:46:53
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
30/05/2023 19:46:53
model ignore: /cache/models/buffalo_l/genderage.onnx genderage
30/05/2023 19:46:53
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
30/05/2023 19:46:54
find model: /cache/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5
30/05/2023 19:46:54
set det-size: (640, 640)
30/05/2023 19:46:54
INFO:     Application startup complete.
30/05/2023 19:46:54
INFO:     Uvicorn running on http://0.0.0.0:3003 (Press CTRL+C to quit)

None of the containers are showing any signs of errors, but the job is stuck at 1. I will let it keep running for a while and see if it does anything.

Alex Tran•2y ago

bring all the containers down them remove redis container after that bring everything backup

QuafleyOP•2y ago

Sure thing, but the redis container is also in the stack so it will take that down automatically.

QuafleyOP•2y ago

Alex Tran•2y ago

I assume so but sometimes the container doesn't necessarily removed so perform docker container prune to make sure

QuafleyOP•2y ago

alright, one moment. Alright, I removed all the images to be sure and restarted the containers. as well as the postgres container

Alex Tran•2y ago

okay now bring them up

QuafleyOP•2y ago

did that, all fine. run recognize faces all again? Nope, still on active 1. Very interesting This is the only thing I see related to the FR job. It simply refuses the execute the FR job, other jobs are executing without problem. I do notice a ton of these error messages when I browse through my photos:

30/05/2023 20:05:38
[Nest] 1  - 05/30/2023, 6:05:38 PM   ERROR [ExpressAdapter] Premature close
30/05/2023 20:05:38
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
30/05/2023 20:05:38
    at new NodeError (node:internal/errors:399:5)
30/05/2023 20:05:38
    at onclose (node:internal/streams/end-of-stream:154:30)
30/05/2023 20:05:38
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11)

30/05/2023 20:05:38
[Nest] 1  - 05/30/2023, 6:05:38 PM   ERROR [ExpressAdapter] Premature close
30/05/2023 20:05:38
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
30/05/2023 20:05:38
    at new NodeError (node:internal/errors:399:5)
30/05/2023 20:05:38
    at onclose (node:internal/streams/end-of-stream:154:30)
30/05/2023 20:05:38
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11)

Alex Tran•2y ago

These are video streaming getting stop/destroy midway and can be ignored

QuafleyOP•2y ago

thank you The compose is the same as reported in here: TL;DR FR job is borked for me and the containers aren't reporting errors even on verbose.

Alex Tran•2y ago

Yeah I am not sure which state the instance is in now @@

QuafleyOP•2y ago

What would you need from me so that we can diagnose the issue? I'm starting to think it is postgres related, however that is also not showing errors. I'm almost at a point of nuking the postgres db and setting up a new instance. Have you guys tested postgres 15?

Alex Tran•2y ago

I am not sure tbh no we haven't

QuafleyOP•2y ago

I am on 15, that is the only thing I can think of

Alex Tran•2y ago

OHHH

QuafleyOP•2y ago

https://tenor.com/view/shook-cant-ne-not-me-who-oop-gif-13118919

Tenor

jrasm91•2y ago

Hmm, nevermind.

QuafleyOP•2y ago

I'm going to start the job with the debug logging on.

jrasm91•2y ago

The delete is this

QuafleyOP•2y ago

I mean, having doubles faces is one. But isn't it weird that the delete is not invoked on job start? seemingly at least

jrasm91•2y ago

That is very strange, because it should be. That's literally what the job is supposed to do.

QuafleyOP•2y ago

It also gives me the heads up will check right now and see what happens LOG_LEVEL=debug

jrasm91•2y ago

Yeah, I'd be interested to see if it shows people were deleted or not.

QuafleyOP•2y ago

Currently the job is also not executing, seemingly since 1.59 oddly enough. yes, same here. hm

QuafleyOP•2y ago

It is not really showing me debug logs. I threw the log_level at the top of the .env file:

LOG_LEVEL=debug
###################################################################################
# Database
###################################################################################

LOG_LEVEL=debug
###################################################################################
# Database
###################################################################################

Why are the containers refusing to give me any information. This is so odd 😂 @jrasm91 Is the recognize all FR working for you on dev 1.59? hey it failed no log of it anywhere hey this is odd

QuafleyOP•2y ago

it is randomly starting to spin up the waiting queue @jrasm91 I finally have a log!

30/05/2023 20:38:30
Request #1685471900713: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
30/05/2023 20:38:30
Request #1685471900713: Sleeping for 4s and then retrying request...
30/05/2023 20:38:34
[Nest] 1  - 05/30/2023, 6:38:34 PM   DEBUG [FacialRecognitionService] Deleted 17260 people and 0 faces

30/05/2023 20:38:30
Request #1685471900713: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
30/05/2023 20:38:30
Request #1685471900713: Sleeping for 4s and then retrying request...
30/05/2023 20:38:34
[Nest] 1  - 05/30/2023, 6:38:34 PM   DEBUG [FacialRecognitionService] Deleted 17260 people and 0 faces

QuafleyOP•2y ago

I may have an idea what is happening. Perhaps it takes so long to delete all faces that the job times out. It would be really nice to get logging on what it is doing instead of just de debug on deletion

jrasm91•2y ago

17k people oh man

QuafleyOP•2y ago

Because it has re-activated itself

QuafleyOP•2y ago

Yes it deleted everyone I think But the weird thing is, I have 14k assets and we had 53k rows within asset_faces. 17k people is insane though I will let this run and check for doubles again.

jrasm91•2y ago

How many photos in your library again?

QuafleyOP•2y ago

40k well 37.7k

QuafleyOP•2y ago

jrasm91•2y ago

Can you send a screenshot from Administration > Server Stats (total usage)

QuafleyOP•2y ago

sure thing

QuafleyOP•2y ago

https://i.imgur.com/4C7S9xK.png

Imgur

jrasm91•2y ago

Why is your queue at 80k

QuafleyOP•2y ago

All my jobs have this

jrasm91•2y ago

x2 assets?

QuafleyOP•2y ago

When I activate a job it goes to double my assets yes

jrasm91•2y ago

Always?

QuafleyOP•2y ago

yes you want me to activate another job to show you? I thought that @Alex mentioned that the doubles assets were normal on jobs?

jrasm91•2y ago

Only on thumbnails That queue generates a webp and a jpeg for each asset.

QuafleyOP•2y ago

Doesn't it generate double assets for you?

jrasm91•2y ago

But it actually only queues all the assets once up front. When an asset finishes with jpeg it then queues another one, so it shouldn't actually ever be that high. Object detection actuall does 2 jobs for each asset right from the start, so that would have 2x the asset count up front.

jrasm91•2y ago

I have 15k assets locally in dev atm. I just ran "all"

QuafleyOP•2y ago

It doesn't do it right away though, after like 5 min sometimes less it is also creating a massive amount of new people

jrasm91•2y ago

Well, queueing all 80k is going to result in duplicates again most likely.

QuafleyOP•2y ago

big chance

jrasm91•2y ago

Can you test on postgres 14?

QuafleyOP•2y ago

Well How? Because I mostlikely can't simply do postgres:15 to postgres:14?

jrasm91•2y ago

No, not with the same volume.

QuafleyOP•2y ago

I will kill the containers either way. So you want me to rebuild?

jrasm91•2y ago

You could do a backup of the data, spin up a 14 database, do a restore, then bring up the rest of the stack https://immich.app/docs/administration/backup-and-restore#database

Backup and Restore | Immich

Database

QuafleyOP•2y ago

good idea let's do that

jrasm91•2y ago

You don't even have to delete the pg volume, you could just point the stack to a new one And switch it back afterwards

QuafleyOP•2y ago

I will first create the backup

jrasm91•2y ago

Yeah, it should be pretty quick.

QuafleyOP•2y ago

alright the dump is done

QuafleyOP•2y ago

that was quick work Okay so now I move postgres to 14? ah wait I see one moment ugh It backed up everything

jrasm91•2y ago

Yeah, I'd just add a new volume to the docker compose file pgdata2 What do you mean?

QuafleyOP•2y ago

These are all the databases

QuafleyOP•2y ago

in 15, I have used freshrss momentarily the other ones I have never touched I could just opt to remove them

jrasm91•2y ago

Ah, you're saying it backup other stuff in addition to immich?

QuafleyOP•2y ago

yes givng me this wild mess

QuafleyOP•2y ago

Alex Tran•2y ago

You probably just need the immich db

QuafleyOP•2y ago

yes How do I change the backup command to only include immich db?

Alex Tran•2y ago

Good question, time for Googling

QuafleyOP•2y ago

rofl

jrasm91•2y ago

Looks like -d <dnname> So -d immich

QuafleyOP•2y ago

docker exec -t postgres_immich pg_dumpall -c -U root -d immich | gzip > "/mnt/user/appdata/immich/dump.sql.gz"

jrasm91•2y ago

look right

QuafleyOP•2y ago

or instead of pg_dumpall? will just test it. hm that is 1kb lol oh wait pg_dumpall > pg_dump that looks better.

jrasm91•2y ago

the command is pg_dumpall

QuafleyOP•2y ago

yes to drop all databases I just want immich right? with pg_dumpall it was 1kb

Alex Tran•2y ago

not to drop, -d indicate the database name

jrasm91•2y ago

Oh, I think that's just what database to connect to originally, it'll still dump all of them.

QuafleyOP•2y ago

oh lol

jrasm91•2y ago

pg_dump does look like it'll do a single database

QuafleyOP•2y ago

this is the answer I got

The command you've given will attempt to dump all databases, not just a specific one, because pg_dumpall is used for dumping all databases. If you only want to dump a specific database (in your case, "immich"), you should use pg_dump instead.

The command you've given will attempt to dump all databases, not just a specific one, because pg_dumpall is used for dumping all databases. If you only want to dump a specific database (in your case, "immich"), you should use pg_dump instead.

jrasm91•2y ago

Sounds good 🙂 TIL

QuafleyOP•2y ago

alright time to gunzip this thing

Alex Tran•2y ago

You asked ChatGPT?

jrasm91•2y ago

Nah, probably a warning/error for using pg_dumpall + -d Or did you?

QuafleyOP•2y ago

restored on 14 Looks identical

jrasm91•2y ago

Now bring up the rest of the stack and try again?

QuafleyOP•2y ago

In this instance I did actually, it gives really well explained answers with links to docs these days. since it has web search function.

jrasm91•2y ago

Dang, thjat's pretty cool.

QuafleyOP•2y ago

It is scary how well it codes :) will see how that goes yes. no apparent errors on startup it seems

QuafleyOP•2y ago

this always takes a bit

Alex Tran•2y ago

yeah same thing for me

jrasm91•2y ago

Typesense is the database server we use for vector queries. It needs to load the asset collection from disk into memory, so it takes longer the bigger the collection is.

QuafleyOP•2y ago

you pushed 1.59.1? I will not update yet

jrasm91•2y ago

Fixed the missing faces job 🙂

Alex Tran•2y ago

that is not out yet

jrasm91•2y ago

Oh right, fixed the broken search page 🙂

QuafleyOP•2y ago

alright I can access the web UI with postgres14

jrasm91•2y ago

Great.

QuafleyOP•2y ago

Lets hit the FR job

QuafleyOP•2y ago

https://tenor.com/view/fingers-crossed-i-hope-so-please-please-gif-19730905

Tenor

QuafleyOP•2y ago

thusfar it stays on 40k

Alex Tran•2y ago

🫣 How did we miss that you were on Postgres15?

QuafleyOP•2y ago

ah shit I've rmoved the logging level in .env I did not think of it before

jrasm91•2y ago

Not a huge deal tbh.

QuafleyOP•2y ago

In the end no, but it shows that 15 is not working as it should

jrasm91•2y ago

Queuing the records 2x explains why you are getting duplicates.

QuafleyOP•2y ago

I've done some janky stuff with this db yes I'm not 100% convinced it works yet, will wait it out.

Alex Tran•2y ago

your CPU must be hot

QuafleyOP•2y ago

yes

QuafleyOP•2y ago

https://tenor.com/view/april-fools-joke-dog-its-fine-this-is-not-gif-16757454

Tenor

QuafleyOP•2y ago

I have messed a lot with this DB already, because I originally used the application from the unraid store. The all in one. The database paths were different. so I had to move in the DB and change the paths With this beauty I believe it was

UPDATE "assets"
SET "originalPath" = 'upload/' || "originalPath",
    "resizePath" = 'upload/' || "resizePath",
    "webpPath" = 'upload/' || "webpPath",
    "encodedVideoPath" = 'upload/' || "encodedVideoPath";

UPDATE "assets"
SET "originalPath" = 'upload/' || "originalPath",
    "resizePath" = 'upload/' || "resizePath",
    "webpPath" = 'upload/' || "webpPath",
    "encodedVideoPath" = 'upload/' || "encodedVideoPath";

The application used '/photos' Okay, so it seem the job is running fine now. Makes me wonder if I should also rerun all other jobs Thank god this is fixed. End result: don't use postgres15 when postgres14 is recommended 🤣

jrasm91•2y ago

haha This is the only job the deletes everything before running. So I think you're OK.

QuafleyOP•2y ago

I presume so as well closed the ticket

Gaming

Programming

Facial recognition double faces, not deleting old ones.

Did you find this page helpful?