Immich•6mo ago

Total images retrieved through pagination doesn't match "total"

Hi everyone, First of all, I want to thank you for this amazing app, it's incredibly useful and works really well! I have a question about API usage. I'm trying to retrieve data for people/faces using the /people endpoint. I currently have 46655 images, so I’m querying them in batches of 1000 (the max allowed). However, the process stops working after approximately 42000 images. Subsequent batches return 0 results, and the hasNextPage field is set to False. To calculate the number of pages, I’m dividing the total number of images by the batch size, i.e., 46655/1000, resulting in 47 batches in total. However, starting from batch 43 (inclusive), I consistently get 0 results. Does anyone know why this is happening? Any insights would be greatly appreciated!

14 Replies

Immich•6mo ago

:wave: Hey @MrMaps, Thanks for reaching out to us. Please follow the recommended actions below; this will help us be more effective in our support effort and leave more time for building Immich :immich:. References - Container Logs: docker compose logs docs - Container Status: docker compose ps docs - Reverse Proxy: https://immich.app/docs/administration/reverse-proxy Checklist 1. :ballot_box_with_check: I have verified I'm on the latest release(note that mobile app releases may take some time). 2. :ballot_box_with_check: I have read applicable release notes. 3. :ballot_box_with_check: I have reviewed the FAQs for known issues. 4. :blue_square: I have reviewed Github for known issues. 5. :ballot_box_with_check: I have tried accessing Immich via local ip (without a custom reverse proxy). 6. :blue_square: I have uploaded the relevant logs, docker compose, and .env files, making sure to use code formatting. 7. :ballot_box_with_check: I have tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable (an item can be marked as "complete" by reacting with the appropriate number) If this ticket can be closed you can use the /close command, and re-open it later if needed.

bo0tzz•6mo ago

Presumably a filter is excluding some assets

MrMapsOP•6mo ago

Do you mean using withHidden? Or what kind of filters? I only have 1 hidden image, but a difference of 5k still seems pretty significant

bo0tzz•6mo ago

Just a general observation, it's impossible to give any specifics without you giving more detail

MrMapsOP•6mo ago

I don't see any relevant data about the requests in the logs of docker, is there any way to check what filters are being used when querying the people endpoint?

immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:17 AM     LOG [Api:RouterExplorer] Mapped {/api/users/profile-image, DELETE} route
immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:17 AM     LOG [Api:RouterExplorer] Mapped {/api/users/:id/profile-image, GET} route
immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:17 AM     LOG [Api:RoutesResolver] ViewController {/api/view}:
immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:17 AM     LOG [Api:RouterExplorer] Mapped {/api/view/folder/unique-paths, GET} route
immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:17 AM     LOG [Api:RouterExplorer] Mapped {/api/view/folder, GET} route
immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:17 AM     LOG [Api:NestApplication] Nest application successfully started
immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:17 AM     LOG [Api:Bootstrap] Immich Server is listening on http://[::1]:2283 [v1.121.0] [production] 
immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:18 AM     LOG [Api:EventRepository] Websocket Connect:    R23yLWzkWhayux0yAAAB
immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:20 AM     LOG [Api:EventRepository] Websocket Connect:    qxobXjTaMR65OOKOAAAD

immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:17 AM     LOG [Api:RouterExplorer] Mapped {/api/users/profile-image, DELETE} route
immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:17 AM     LOG [Api:RouterExplorer] Mapped {/api/users/:id/profile-image, GET} route
immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:17 AM     LOG [Api:RoutesResolver] ViewController {/api/view}:
immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:17 AM     LOG [Api:RouterExplorer] Mapped {/api/view/folder/unique-paths, GET} route
immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:17 AM     LOG [Api:RouterExplorer] Mapped {/api/view/folder, GET} route
immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:17 AM     LOG [Api:NestApplication] Nest application successfully started
immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:17 AM     LOG [Api:Bootstrap] Immich Server is listening on http://[::1]:2283 [v1.121.0] [production] 
immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:18 AM     LOG [Api:EventRepository] Websocket Connect:    R23yLWzkWhayux0yAAAB
immich-server-1  | [Nest] 15  - 11/26/2024, 11:14:20 AM     LOG [Api:EventRepository] Websocket Connect:    qxobXjTaMR65OOKOAAAD

bo0tzz•6mo ago

Do you have the log level set to verbose?

MrMapsOP•6mo ago

I'll start from that, thank you

Daniel•6mo ago

Could you be more precise about which requests you're making in the first place? You were saying you make requests against /people, but then you also talk about total asset count and I am confused

MrMapsOP•6mo ago

Sorry, I'm referring to the "total" field you get when querying the /people endpoint

 def retrieve_unknown_faces_paginated(page, result):
     res = req.get(
         "https://domain/api/people",
         params={
             "page": page,
             "size": MAX,
         },
         headers=ut.HEADERS["immich"],
         timeout=120,
     )
     data = res.json()
     print(data)
     # {'people': [], 'hasNextPage': False, 'total': 46655, 'hidden': 1}

 def retrieve_unknown_faces_paginated(page, result):
     res = req.get(
         "https://domain/api/people",
         params={
             "page": page,
             "size": MAX,
         },
         headers=ut.HEADERS["immich"],
         timeout=120,
     )
     data = res.json()
     print(data)
     # {'people': [], 'hasNextPage': False, 'total': 46655, 'hidden': 1}

Above is the request for page 43, with size 1000. This is normal request (page 0):

{'people': [{'id': '95478b5b-db69-4dc1-87b8-319176898d0d', 'name': '', 'birthDate': None, 'thumbnailPath': 'upload/thumbs/90f33fc8-629b-48d4-be3d-11a9a6e112de/95/47/95478b5b-db69-4dc1-87b8-319176898d0d.jpeg', 'isHidden': False, 'updatedAt': '2024-10-31T15:34:59.734Z'}, ..., {'id': '6c4a4ddb-a143-48bf-980e-115352781d8e', 'name': '', 'birthDate': None, 'thumbnailPath': 'upload/thumbs/90f33fc8-629b-48d4-be3d-11a9a6e112de/6c/4a/6c4a4ddb-a143-48bf-980e-115352781d8e.jpeg', 'isHidden': False, 'updatedAt': '2024-10-31T18:53:22.499Z'}], 'hasNextPage': True, 'total': 46655, 'hidden': 1}

{'people': [{'id': '95478b5b-db69-4dc1-87b8-319176898d0d', 'name': '', 'birthDate': None, 'thumbnailPath': 'upload/thumbs/90f33fc8-629b-48d4-be3d-11a9a6e112de/95/47/95478b5b-db69-4dc1-87b8-319176898d0d.jpeg', 'isHidden': False, 'updatedAt': '2024-10-31T15:34:59.734Z'}, ..., {'id': '6c4a4ddb-a143-48bf-980e-115352781d8e', 'name': '', 'birthDate': None, 'thumbnailPath': 'upload/thumbs/90f33fc8-629b-48d4-be3d-11a9a6e112de/6c/4a/6c4a4ddb-a143-48bf-980e-115352781d8e.jpeg', 'isHidden': False, 'updatedAt': '2024-10-31T18:53:22.499Z'}], 'hasNextPage': True, 'total': 46655, 'hidden': 1}

Daniel•6mo ago

Oh I see That's a known bug actually The total count does not take the minimum amount of faces into account #13245

Immich•6mo ago

[Issue] People count doesn't respect minimum recognized faces (immich-app/immich#13245)

MrMapsOP•6mo ago

Thank you very much for the insights! I'll switch to use the hasNextPage field instead of relying in the total counter.

Daniel•6mo ago

That's generally always a good idea when using paginated endpoints :)

Immich•6mo ago

This thread has been closed. To re-open, use the button below.

Gaming

Programming

Total images retrieved through pagination doesn't match "total"

Did you find this page helpful?