Immich 1.133.0 Upgrade Issues from 1.132.0

I am running Immich and Immich ML on a docker-compose with a separate dedicated postgres instance as I have other services using that database server. When upgrading immich without upgrading the database server, it fails to start. If I attempt to upgrade postgres to the latest recommended container, it fails to start there as well. I have moved to valkey for cache server sometime ago even though the domain says redis. See below for more technical information. Configuration files: Immich compose:
services:
immich-server:
image: ghcr.io/immich-app/immich-server:release
container_name: immich-server
restart: unless-stopped
ports:
- 2283:2283
volumes:
- ${UPLOAD_PATH}:/usr/src/app/upload
- /etc/localtime:/etc/localtime:ro
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities:
- gpu
- compute
- utility
- video
healthcheck:
disable: false
user: 1000:1000
env_file:
- .env
immich-ml:
image: ghcr.io/immich-app/immich-machine-learning:release-cuda
container_name: immich-ml
restart: unless-stopped
volumes:
- ${CACHE_PATH}:/cache
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities:
- gpu
- compute
- utility
- video
healthcheck:
disable: false
user: 1000:1000
env_file:
- .env
services:
immich-server:
image: ghcr.io/immich-app/immich-server:release
container_name: immich-server
restart: unless-stopped
ports:
- 2283:2283
volumes:
- ${UPLOAD_PATH}:/usr/src/app/upload
- /etc/localtime:/etc/localtime:ro
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities:
- gpu
- compute
- utility
- video
healthcheck:
disable: false
user: 1000:1000
env_file:
- .env
immich-ml:
image: ghcr.io/immich-app/immich-machine-learning:release-cuda
container_name: immich-ml
restart: unless-stopped
volumes:
- ${CACHE_PATH}:/cache
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities:
- gpu
- compute
- utility
- video
healthcheck:
disable: false
user: 1000:1000
env_file:
- .env
.env:
# App
UPLOAD_PATH=/mnt/media/immich
CACHE_PATH=/mnt/data/immich
TZ=US/Eastern
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_DRIVER_CAPABILITIES=all

# Database
DB_USERNAME=immich
DB_PASSWORD=<change-me>
DB_DATABASE_NAME=immich
DB_HOSTNAME=postgres.apps.domain.cc
DB_PORT=5432
DB_VECTOR_EXTENSION=pgvecto.rs

# Caching
REDIS_HOSTNAME=redis.apps.domain.cc
REDIS_PORT=6379
# App
UPLOAD_PATH=/mnt/media/immich
CACHE_PATH=/mnt/data/immich
TZ=US/Eastern
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_DRIVER_CAPABILITIES=all

# Database
DB_USERNAME=immich
DB_PASSWORD=<change-me>
DB_DATABASE_NAME=immich
DB_HOSTNAME=postgres.apps.domain.cc
DB_PORT=5432
DB_VECTOR_EXTENSION=pgvecto.rs

# Caching
REDIS_HOSTNAME=redis.apps.domain.cc
REDIS_PORT=6379
93 Replies
Immich
Immich2w ago
:wave: Hey @dasunsrule32, Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:. References - Container Logs: docker compose logs docs - Container Status: docker ps -a docs - Reverse Proxy: https://immich.app/docs/administration/reverse-proxy - Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA Checklist I have... 1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time). 2. :blue_square: read applicable release notes. 3. :blue_square: reviewed the FAQs for known issues. 4. :blue_square: reviewed Github for known issues. 5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy). 6. :blue_square: uploaded the relevant information (see below). 7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable (an item can be marked as "complete" by reacting with the appropriate number) Information In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider: - Your docker-compose.yml and .env files. - Logs from all the containers and their status (see above). - All the troubleshooting steps you've tried so far. - Any recent changes you've made to Immich or your system. - Details about your system (both software/OS and hardware). - Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h). - The version of the Immich server, mobile app, and other relevant pieces. - Any other information that you think might be relevant. Please paste files and logs with proper code formatting, and especially avoid blurry screenshots. Without the right information we can't work out what the problem is. Help us help you ;) If this ticket can be closed you can use the /close command, and re-open it later if needed.
dasunsrule32
dasunsrule32OP2w ago
Postgres compose:
services:
postgres:
image: tensorchord/pgvecto-rs:pg16-v0.3.0
#image: ghcr.io/immich-app/postgres:16-vectorchord0.3.0-pgvectors0.3.0
container_name: postgres
restart: unless-stopped
ports:
- 5432:5432
volumes:
- ${DB_PATH}:/var/lib/postgresql/data
healthcheck:
test:
- CMD-SHELL
- pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}
start_period: 20s
interval: 30s
retries: 5
timeout: 5s
deploy:
resources:
limits:
memory: 1G
user: 1000:1000
env_file:
- .env
services:
postgres:
image: tensorchord/pgvecto-rs:pg16-v0.3.0
#image: ghcr.io/immich-app/postgres:16-vectorchord0.3.0-pgvectors0.3.0
container_name: postgres
restart: unless-stopped
ports:
- 5432:5432
volumes:
- ${DB_PATH}:/var/lib/postgresql/data
healthcheck:
test:
- CMD-SHELL
- pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}
start_period: 20s
interval: 30s
retries: 5
timeout: 5s
deploy:
resources:
limits:
memory: 1G
user: 1000:1000
env_file:
- .env
Postgres .env:
# postgres App
DB_PATH=/mnt/db/postgres
PGDATA=/var/lib/postgresql/data/pgdata

# Database
POSTGRES_DB=postgres
POSTGRES_USER=admin
POSTGRES_PASSWORD=<change-me>
# postgres App
DB_PATH=/mnt/db/postgres
PGDATA=/var/lib/postgresql/data/pgdata

# Database
POSTGRES_DB=postgres
POSTGRES_USER=admin
POSTGRES_PASSWORD=<change-me>
Getting errors/warnings in the logs:
[Nest] 7 - 05/22/2025, 10:53:15 AM WARN [Microservices:DatabaseService] Could not run vector reindexing checks. If the extension was updated, please restart the Postgres instance. If you are upgrading directly from a version below 1.107.2, please upgrade to 1.107.2 first.
PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
[Nest] 7 - 05/22/2025, 10:53:15 AM WARN [Microservices:DatabaseService] Could not run vector reindexing checks. If the extension was updated, please restart the Postgres instance. If you are upgrading directly from a version below 1.107.2, please upgrade to 1.107.2 first.
PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
I checked the server and nothing has changed there.
Zack
Zack2w ago
Can you post all of the logs please?
dasunsrule32
dasunsrule32OP2w ago
Keep running into char counts limits
immich-server | [Nest] 7 - 05/22/2025, 11:04:59 AM WARN [Microservices:DatabaseService] Could not run vector reindexing checks. If the extension was updated, please restart the Postgres instance. If you are upgrading directly from a version below 1.107.2, please upgrade to 1.107.2 first.
immich-server | PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
immich-server | ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`
immich-server | at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
immich-server | at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
immich-server | at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
immich-server | at Socket.emit (node:events:518:28)
immich-server | at addChunk (node:internal/streams/readable:561:12)
immich-server | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
immich-server | at Readable.push (node:internal/streams/readable:392:5)
immich-server | at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
immich-server | severity_local: 'ERROR',
immich-server | severity: 'ERROR',
immich-server | code: 'XX000',
immich-server | file: 'error.rs',
immich-server | line: '125',
immich-server | routine: 'vectors::error::check_client'
immich-server | }
immich-server | [Nest] 7 - 05/22/2025, 11:04:59 AM WARN [Microservices:DatabaseService] Could not run vector reindexing checks. If the extension was updated, please restart the Postgres instance. If you are upgrading directly from a version below 1.107.2, please upgrade to 1.107.2 first.
immich-server | PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
immich-server | ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`
immich-server | at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
immich-server | at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
immich-server | at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
immich-server | at Socket.emit (node:events:518:28)
immich-server | at addChunk (node:internal/streams/readable:561:12)
immich-server | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
immich-server | at Readable.push (node:internal/streams/readable:392:5)
immich-server | at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
immich-server | severity_local: 'ERROR',
immich-server | severity: 'ERROR',
immich-server | code: 'XX000',
immich-server | file: 'error.rs',
immich-server | line: '125',
immich-server | routine: 'vectors::error::check_client'
immich-server | }
immich-server | microservices worker error: PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
immich-server | ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`, stack: PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
immich-server | ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`
immich-server | at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
immich-server | at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
immich-server | at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
immich-server | at Socket.emit (node:events:518:28)
immich-server | at addChunk (node:internal/streams/readable:561:12)
immich-server | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
immich-server | at Readable.push (node:internal/streams/readable:392:5)
immich-server | at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
immich-server | microservices worker exited with code 1
immich-server | Killing api process
immich-server | microservices worker error: PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
immich-server | ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`, stack: PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
immich-server | ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`
immich-server | at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
immich-server | at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
immich-server | at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
immich-server | at Socket.emit (node:events:518:28)
immich-server | at addChunk (node:internal/streams/readable:561:12)
immich-server | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
immich-server | at Readable.push (node:internal/streams/readable:392:5)
immich-server | at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
immich-server | microservices worker exited with code 1
immich-server | Killing api process
immich-server | Initializing Immich v1.133.0
immich-server | Detected CPU Cores: 16
immich-server | Starting api worker
immich-server | Starting microservices worker
immich-server | [Nest] 7 - 05/22/2025, 11:06:04 AM LOG [Microservices:EventRepository] Initialized websocket server
immich-server | [Nest] 7 - 05/22/2025, 11:06:04 AM LOG [Microservices:DatabaseRepository] Reindexing clip_index
immich-server | [Nest] 7 - 05/22/2025, 11:06:04 AM LOG [Microservices:DatabaseRepository] Reindexing face_index
immich-server | Initializing Immich v1.133.0
immich-server | Detected CPU Cores: 16
immich-server | Starting api worker
immich-server | Starting microservices worker
immich-server | [Nest] 7 - 05/22/2025, 11:06:04 AM LOG [Microservices:EventRepository] Initialized websocket server
immich-server | [Nest] 7 - 05/22/2025, 11:06:04 AM LOG [Microservices:DatabaseRepository] Reindexing clip_index
immich-server | [Nest] 7 - 05/22/2025, 11:06:04 AM LOG [Microservices:DatabaseRepository] Reindexing face_index
immich-server | Query failed : {
immich-server | durationMs: 2.2855659999995623,
immich-server | error: PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
immich-server | ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`
immich-server | at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
immich-server | at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
immich-server | at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
immich-server | at Socket.emit (node:events:518:28)
immich-server | at addChunk (node:internal/streams/readable:561:12)
immich-server | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
immich-server | at Readable.push (node:internal/streams/readable:392:5)
immich-server | at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
immich-server | severity_local: 'ERROR',
immich-server | severity: 'ERROR',
immich-server | code: 'XX000',
immich-server | file: 'error.rs',
immich-server | line: '125',
immich-server | routine: 'vectors::error::check_client'
immich-server | },
immich-server | sql: '\n' +
immich-server | ' CREATE INDEX IF NOT EXISTS clip_index ON smart_search\n' +
immich-server | ' USING vectors (embedding vector_cos_ops) WITH (options = $$\n' +
immich-server | ' optimizing.optimizing_threads = 4\n' +
immich-server | ' [indexing.hnsw]\n' +
immich-server | ' m = 16\n' +
immich-server | ' ef_construction = 300\n' +
immich-server | ' $$)',
immich-server | params: []
immich-server | }
immich-server | Query failed : {
immich-server | durationMs: 2.2855659999995623,
immich-server | error: PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
immich-server | ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`
immich-server | at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
immich-server | at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
immich-server | at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
immich-server | at Socket.emit (node:events:518:28)
immich-server | at addChunk (node:internal/streams/readable:561:12)
immich-server | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
immich-server | at Readable.push (node:internal/streams/readable:392:5)
immich-server | at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
immich-server | severity_local: 'ERROR',
immich-server | severity: 'ERROR',
immich-server | code: 'XX000',
immich-server | file: 'error.rs',
immich-server | line: '125',
immich-server | routine: 'vectors::error::check_client'
immich-server | },
immich-server | sql: '\n' +
immich-server | ' CREATE INDEX IF NOT EXISTS clip_index ON smart_search\n' +
immich-server | ' USING vectors (embedding vector_cos_ops) WITH (options = $$\n' +
immich-server | ' optimizing.optimizing_threads = 4\n' +
immich-server | ' [indexing.hnsw]\n' +
immich-server | ' m = 16\n' +
immich-server | ' ef_construction = 300\n' +
immich-server | ' $$)',
immich-server | params: []
immich-server | }
immich-server | [Nest] 7 - 05/22/2025, 11:06:04 AM WARN [Microservices:DatabaseService] Could not run vector reindexing checks. If the extension was updated, please restart the Postgres instance. If you are upgrading directly from a version below 1.107.2, please upgrade to 1.107.2 first.
immich-server | PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
immich-server | ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`
immich-server | at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
immich-server | at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
immich-server | at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
immich-server | at Socket.emit (node:events:518:28)
immich-server | at addChunk (node:internal/streams/readable:561:12)
immich-server | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
immich-server | at Readable.push (node:internal/streams/readable:392:5)
immich-server | at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
immich-server | severity_local: 'ERROR',
immich-server | severity: 'ERROR',
immich-server | code: 'XX000',
immich-server | file: 'error.rs',
immich-server | line: '125',
immich-server | routine: 'vectors::error::check_client'
immich-server | }
immich-server | [Nest] 7 - 05/22/2025, 11:06:04 AM WARN [Microservices:DatabaseService] Could not run vector reindexing checks. If the extension was updated, please restart the Postgres instance. If you are upgrading directly from a version below 1.107.2, please upgrade to 1.107.2 first.
immich-server | PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
immich-server | ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`
immich-server | at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
immich-server | at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
immich-server | at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
immich-server | at Socket.emit (node:events:518:28)
immich-server | at addChunk (node:internal/streams/readable:561:12)
immich-server | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
immich-server | at Readable.push (node:internal/streams/readable:392:5)
immich-server | at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
immich-server | severity_local: 'ERROR',
immich-server | severity: 'ERROR',
immich-server | code: 'XX000',
immich-server | file: 'error.rs',
immich-server | line: '125',
immich-server | routine: 'vectors::error::check_client'
immich-server | }
immich-server | microservices worker error: PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
immich-server | ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`, stack: PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
immich-server | ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`
immich-server | at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
immich-server | at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
immich-server | at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
immich-server | at Socket.emit (node:events:518:28)
immich-server | at addChunk (node:internal/streams/readable:561:12)
immich-server | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
immich-server | at Readable.push (node:internal/streams/readable:392:5)
immich-server | at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
immich-server | microservices worker exited with code 1
immich-server | Killing api process
immich-server | microservices worker error: PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
immich-server | ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`, stack: PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
immich-server | ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`
immich-server | at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
immich-server | at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
immich-server | at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
immich-server | at Socket.emit (node:events:518:28)
immich-server | at addChunk (node:internal/streams/readable:561:12)
immich-server | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
immich-server | at Readable.push (node:internal/streams/readable:392:5)
immich-server | at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
immich-server | microservices worker exited with code 1
immich-server | Killing api process
This is without upgrading the postgres server
Zack
Zack2w ago
Did you update your database image from 0.2.0 or something?
dasunsrule32
dasunsrule32OP2w ago
image: tensorchord/pgvecto-rs:pg16-v0.3.0 I've been running that one for awhile
Zack
Zack2w ago
I saw that, but pgvecto.rs seems to think your indexes are from an old version What was your reason for not wanting to switch to the new image btw?
dasunsrule32
dasunsrule32OP2w ago
I did nothing other than upgrade from 1.132.0 to 1.133.0 I tried, nothing logs in on the new postgres image. I can't access it when I cutover
Zack
Zack2w ago
Do you get these same errors when trying to upgrade to the new image?
dasunsrule32
dasunsrule32OP2w ago
My app accounts, admin accounts, etc. I tried pulling it again and it's failing to start.
Zack
Zack2w ago
Could you send the logs for that? To avoid char limits you can make a file with the logs, then send that here
dasunsrule32
dasunsrule32OP2w ago
Yeah, one sec, switching to it now
postgres | Using SSD storage
postgres | chmod: changing permissions of '/var/run/postgresql': Operation not permitted
postgres |
postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres |
postgres | 2025-05-22 15:12:46.333 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.conf"
postgres | 2025-05-22 15:12:46.333 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
postgres | 2025-05-22 15:12:46.333 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.conf"
postgres | 2025-05-22 15:12:46.333 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
postgres | 2025-05-22 15:12:47.765 GMT [1] LOG: starting PostgreSQL 16.8 (Debian 16.8-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc
postgres | 2025-05-22 15:12:47.766 GMT [1] LOG: listening on IPv6 address "::1", port 5432
postgres | 2025-05-22 15:12:47.766 GMT [1] LOG: listening on IPv4 address "127.0.0.1", port 5432
postgres | 2025-05-22 15:12:47.832 GMT [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres | 2025-05-22 15:12:47.847 GMT [17] LOG: database system was shut down at 2025-05-22 15:12:41 GMT
postgres | 2025-05-22 15:12:47.883 GMT [1] LOG: database system is ready to accept connections
postgres | Using SSD storage
postgres | chmod: changing permissions of '/var/run/postgresql': Operation not permitted
postgres |
postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres |
postgres | 2025-05-22 15:12:46.333 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.conf"
postgres | 2025-05-22 15:12:46.333 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
postgres | 2025-05-22 15:12:46.333 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.conf"
postgres | 2025-05-22 15:12:46.333 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
postgres | 2025-05-22 15:12:47.765 GMT [1] LOG: starting PostgreSQL 16.8 (Debian 16.8-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc
postgres | 2025-05-22 15:12:47.766 GMT [1] LOG: listening on IPv6 address "::1", port 5432
postgres | 2025-05-22 15:12:47.766 GMT [1] LOG: listening on IPv4 address "127.0.0.1", port 5432
postgres | 2025-05-22 15:12:47.832 GMT [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres | 2025-05-22 15:12:47.847 GMT [17] LOG: database system was shut down at 2025-05-22 15:12:41 GMT
postgres | 2025-05-22 15:12:47.883 GMT [1] LOG: database system is ready to accept connections
Zack
Zack2w ago
That looks like it started just fine
dasunsrule32
dasunsrule32OP2w ago
It's not accepting anything
Zack
Zack2w ago
Although you have some weird config issues Hmmm
dasunsrule32
dasunsrule32OP2w ago
No app can connect I drop back to the other server and everything comes back up
dasunsrule32
dasunsrule32OP2w ago
No description
dasunsrule32
dasunsrule32OP2w ago
Here is pgadmin on the new image
Zack
Zack2w ago
Hmmm One minute, I might know what's going on there
dasunsrule32
dasunsrule32OP2w ago
My authentik instance is offline as well, anything that logs in can't Ok
Zack
Zack2w ago
How is pgadmin configured to talk to the database? Like, the compose network setup
dasunsrule32
dasunsrule32OP2w ago
One sec
pgadmin:
image: dpage/pgadmin4:latest
container_name: pgadmin
restart: unless-stopped
ports:
- 80:80
volumes:
- ${DATA_PATH}:/var/lib/pgadmin
- ${CONFIG_PATH}:/pgadmin4/config_local.py
deploy:
resources:
limits:
memory: 1G
depends_on:
postgres:
condition: service_healthy
env_file:
- .env
pgadmin:
image: dpage/pgadmin4:latest
container_name: pgadmin
restart: unless-stopped
ports:
- 80:80
volumes:
- ${DATA_PATH}:/var/lib/pgadmin
- ${CONFIG_PATH}:/pgadmin4/config_local.py
deploy:
resources:
limits:
memory: 1G
depends_on:
postgres:
condition: service_healthy
env_file:
- .env
Zack
Zack2w ago
This is in the same compose file as the database?
dasunsrule32
dasunsrule32OP2w ago
# pgAdmin App
DATA_PATH=/mnt/data/pgadmin
CONFIG_PATH=/mnt/data/pgadmin/config_local.py
PGADMIN_DISABLE_POSTFIX=true
# pgAdmin App
DATA_PATH=/mnt/data/pgadmin
CONFIG_PATH=/mnt/data/pgadmin/config_local.py
PGADMIN_DISABLE_POSTFIX=true
Yes config:
WTF_CSRF_CHECK_DEFAULT = False
SERVER_MODE = True
MASTER_PASSWORD = True
AUTHENTICATION_SOURCES = ['oauth2', 'internal']
OAUTH2_AUTO_CREATE_USER = True
OAUTH2_CONFIG = [{
'OAUTH2_NAME': 'oidc',
'OAUTH2_DISPLAY_NAME': 'ID',
'OAUTH2_CLIENT_ID': 'pgadmin',
'OAUTH2_CLIENT_SECRET': '***',
'OAUTH2_TOKEN_URL': 'https://auth.domain.co/application/o/token/',
'OAUTH2_AUTHORIZATION_URL': 'https://auth.domain.co/application/o/authorize/',
'OAUTH2_API_BASE_URL': 'https://auth.domain.co/application/o/pgadmin/',
'OAUTH2_SERVER_METADATA_URL': 'https://auth.domain.co/application/o/pgadmin/.well-known/openid-configuration',
'OAUTH2_USERINFO_ENDPOINT': 'https://auth.domain.co/application/o/userinfo/',
'OAUTH2_SCOPE': 'openid email profile',
'OAUTH2_ICON' : 'fa-houzz',
'OAUTH2_BUTTON_COLOR': '#F4253E',
'OAUTH2_USERNAME_CLAIM': 'email',
'OAUTH2_ADDITIONAL_CLAIMS': {
'groups': ["pgAdmin Admins"],
}
}]
WTF_CSRF_CHECK_DEFAULT = False
SERVER_MODE = True
MASTER_PASSWORD = True
AUTHENTICATION_SOURCES = ['oauth2', 'internal']
OAUTH2_AUTO_CREATE_USER = True
OAUTH2_CONFIG = [{
'OAUTH2_NAME': 'oidc',
'OAUTH2_DISPLAY_NAME': 'ID',
'OAUTH2_CLIENT_ID': 'pgadmin',
'OAUTH2_CLIENT_SECRET': '***',
'OAUTH2_TOKEN_URL': 'https://auth.domain.co/application/o/token/',
'OAUTH2_AUTHORIZATION_URL': 'https://auth.domain.co/application/o/authorize/',
'OAUTH2_API_BASE_URL': 'https://auth.domain.co/application/o/pgadmin/',
'OAUTH2_SERVER_METADATA_URL': 'https://auth.domain.co/application/o/pgadmin/.well-known/openid-configuration',
'OAUTH2_USERINFO_ENDPOINT': 'https://auth.domain.co/application/o/userinfo/',
'OAUTH2_SCOPE': 'openid email profile',
'OAUTH2_ICON' : 'fa-houzz',
'OAUTH2_BUTTON_COLOR': '#F4253E',
'OAUTH2_USERNAME_CLAIM': 'email',
'OAUTH2_ADDITIONAL_CLAIMS': {
'groups': ["pgAdmin Admins"],
}
}]
I added the server manually in pgadmin
Zack
Zack2w ago
postgres | 2025-05-22 15:12:46.333 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.conf" This is the line that confuses me This should absolutely exist Can you check your postgres data directory and see if it exists there
Zack
Zack2w ago
It should look similar to this
No description
dasunsrule32
dasunsrule32OP2w ago
I have no name!@dfd2c64982e4:/var/lib/postgresql$ ls -la */*
total 106
drwx------ 21 1000 1000 28 May 22 15:12 .
drwx------ 3 1000 1000 3 Apr 17 2024 ..
drwx------ 14 1000 1000 14 May 22 14:26 base
drwx------ 2 1000 1000 68 May 22 15:13 global
drwx------ 2 1000 1000 359 May 22 14:40 log
drwx------ 2 1000 1000 2 Apr 17 2024 pg_commit_ts
drwx------ 2 1000 1000 2 Apr 17 2024 pg_dynshmem
-rw------- 1 1000 1000 5743 Apr 17 2024 pg_hba.conf
-rw------- 1 1000 1000 2640 Apr 17 2024 pg_ident.conf
drwx------ 4 1000 1000 5 May 22 15:17 pg_logical
drwx------ 4 1000 1000 4 Apr 17 2024 pg_multixact
drwx------ 2 1000 1000 2 Apr 17 2024 pg_notify
drwx------ 2 1000 1000 2 Apr 17 2024 pg_replslot
drwx------ 2 1000 1000 2 Apr 17 2024 pg_serial
drwx------ 2 1000 1000 2 Apr 17 2024 pg_snapshots
drwx------ 2 1000 1000 2 May 22 15:12 pg_stat
drwx------ 2 1000 1000 2 Apr 17 2024 pg_stat_tmp
drwx------ 2 1000 1000 3 May 21 16:52 pg_subtrans
drwx------ 2 1000 1000 2 Apr 17 2024 pg_tblspc
drwx------ 2 1000 1000 2 Apr 17 2024 pg_twophase
drwx------ 4 1000 1000 5 Jan 2 16:29 pg_vectors
-rw------- 1 1000 1000 3 Apr 17 2024 PG_VERSION
drwx------ 3 1000 1000 11 May 22 15:12 pg_wal
drwx------ 2 1000 1000 17 May 11 12:27 pg_xact
-rw------- 1 1000 1000 88 Apr 17 2024 postgresql.auto.conf
-rw------- 1 1000 1000 29770 Apr 17 2024 postgresql.conf
-rw------- 1 1000 1000 87 May 22 15:12 postmaster.opts
-rw------- 1 1000 1000 109 May 22 15:12 postmaster.pid
I have no name!@dfd2c64982e4:/var/lib/postgresql$ ls -la */*
total 106
drwx------ 21 1000 1000 28 May 22 15:12 .
drwx------ 3 1000 1000 3 Apr 17 2024 ..
drwx------ 14 1000 1000 14 May 22 14:26 base
drwx------ 2 1000 1000 68 May 22 15:13 global
drwx------ 2 1000 1000 359 May 22 14:40 log
drwx------ 2 1000 1000 2 Apr 17 2024 pg_commit_ts
drwx------ 2 1000 1000 2 Apr 17 2024 pg_dynshmem
-rw------- 1 1000 1000 5743 Apr 17 2024 pg_hba.conf
-rw------- 1 1000 1000 2640 Apr 17 2024 pg_ident.conf
drwx------ 4 1000 1000 5 May 22 15:17 pg_logical
drwx------ 4 1000 1000 4 Apr 17 2024 pg_multixact
drwx------ 2 1000 1000 2 Apr 17 2024 pg_notify
drwx------ 2 1000 1000 2 Apr 17 2024 pg_replslot
drwx------ 2 1000 1000 2 Apr 17 2024 pg_serial
drwx------ 2 1000 1000 2 Apr 17 2024 pg_snapshots
drwx------ 2 1000 1000 2 May 22 15:12 pg_stat
drwx------ 2 1000 1000 2 Apr 17 2024 pg_stat_tmp
drwx------ 2 1000 1000 3 May 21 16:52 pg_subtrans
drwx------ 2 1000 1000 2 Apr 17 2024 pg_tblspc
drwx------ 2 1000 1000 2 Apr 17 2024 pg_twophase
drwx------ 4 1000 1000 5 Jan 2 16:29 pg_vectors
-rw------- 1 1000 1000 3 Apr 17 2024 PG_VERSION
drwx------ 3 1000 1000 11 May 22 15:12 pg_wal
drwx------ 2 1000 1000 17 May 11 12:27 pg_xact
-rw------- 1 1000 1000 88 Apr 17 2024 postgresql.auto.conf
-rw------- 1 1000 1000 29770 Apr 17 2024 postgresql.conf
-rw------- 1 1000 1000 87 May 22 15:12 postmaster.opts
-rw------- 1 1000 1000 109 May 22 15:12 postmaster.pid
I have no name!@dfd2c64982e4:/var/lib/postgresql$ id
uid=1000 gid=1000 groups=1000
I have no name!@dfd2c64982e4:/var/lib/postgresql$ id
uid=1000 gid=1000 groups=1000
Zack
Zack2w ago
It exists 🤔 Wait, that's in /var/lib/postgresql Is that the location on the host?
dasunsrule32
dasunsrule32OP2w ago
That's in the container docker exec -it postgres /bin/bash
Zack
Zack2w ago
volumes:
- ${DB_PATH}:/var/lib/postgresql/data
volumes:
- ${DB_PATH}:/var/lib/postgresql/data
This should be mounted to /var/lib/postgresql/data Is that first line truncated in your console output?
dasunsrule32
dasunsrule32OP2w ago
Yeah
I have no name!@dfd2c64982e4:/var/lib/postgresql/data/pgdata$ ls -la
total 106
drwx------ 21 1000 1000 28 May 22 15:12 .
drwx------ 3 1000 1000 3 Apr 17 2024 ..
drwx------ 14 1000 1000 14 May 22 14:26 base
drwx------ 2 1000 1000 68 May 22 15:13 global
drwx------ 2 1000 1000 359 May 22 14:40 log
drwx------ 2 1000 1000 2 Apr 17 2024 pg_commit_ts
drwx------ 2 1000 1000 2 Apr 17 2024 pg_dynshmem
-rw------- 1 1000 1000 5743 Apr 17 2024 pg_hba.conf
-rw------- 1 1000 1000 2640 Apr 17 2024 pg_ident.conf
drwx------ 4 1000 1000 5 May 22 15:17 pg_logical
drwx------ 4 1000 1000 4 Apr 17 2024 pg_multixact
drwx------ 2 1000 1000 2 Apr 17 2024 pg_notify
drwx------ 2 1000 1000 2 Apr 17 2024 pg_replslot
drwx------ 2 1000 1000 2 Apr 17 2024 pg_serial
drwx------ 2 1000 1000 2 Apr 17 2024 pg_snapshots
drwx------ 2 1000 1000 2 May 22 15:12 pg_stat
drwx------ 2 1000 1000 2 Apr 17 2024 pg_stat_tmp
drwx------ 2 1000 1000 3 May 21 16:52 pg_subtrans
drwx------ 2 1000 1000 2 Apr 17 2024 pg_tblspc
drwx------ 2 1000 1000 2 Apr 17 2024 pg_twophase
drwx------ 4 1000 1000 5 Jan 2 16:29 pg_vectors
-rw------- 1 1000 1000 3 Apr 17 2024 PG_VERSION
drwx------ 3 1000 1000 11 May 22 15:12 pg_wal
drwx------ 2 1000 1000 17 May 11 12:27 pg_xact
-rw------- 1 1000 1000 88 Apr 17 2024 postgresql.auto.conf
-rw------- 1 1000 1000 29770 Apr 17 2024 postgresql.conf
-rw------- 1 1000 1000 87 May 22 15:12 postmaster.opts
-rw------- 1 1000 1000 109 May 22 15:12 postmaster.pid
I have no name!@dfd2c64982e4:/var/lib/postgresql/data/pgdata$ ls -la
total 106
drwx------ 21 1000 1000 28 May 22 15:12 .
drwx------ 3 1000 1000 3 Apr 17 2024 ..
drwx------ 14 1000 1000 14 May 22 14:26 base
drwx------ 2 1000 1000 68 May 22 15:13 global
drwx------ 2 1000 1000 359 May 22 14:40 log
drwx------ 2 1000 1000 2 Apr 17 2024 pg_commit_ts
drwx------ 2 1000 1000 2 Apr 17 2024 pg_dynshmem
-rw------- 1 1000 1000 5743 Apr 17 2024 pg_hba.conf
-rw------- 1 1000 1000 2640 Apr 17 2024 pg_ident.conf
drwx------ 4 1000 1000 5 May 22 15:17 pg_logical
drwx------ 4 1000 1000 4 Apr 17 2024 pg_multixact
drwx------ 2 1000 1000 2 Apr 17 2024 pg_notify
drwx------ 2 1000 1000 2 Apr 17 2024 pg_replslot
drwx------ 2 1000 1000 2 Apr 17 2024 pg_serial
drwx------ 2 1000 1000 2 Apr 17 2024 pg_snapshots
drwx------ 2 1000 1000 2 May 22 15:12 pg_stat
drwx------ 2 1000 1000 2 Apr 17 2024 pg_stat_tmp
drwx------ 2 1000 1000 3 May 21 16:52 pg_subtrans
drwx------ 2 1000 1000 2 Apr 17 2024 pg_tblspc
drwx------ 2 1000 1000 2 Apr 17 2024 pg_twophase
drwx------ 4 1000 1000 5 Jan 2 16:29 pg_vectors
-rw------- 1 1000 1000 3 Apr 17 2024 PG_VERSION
drwx------ 3 1000 1000 11 May 22 15:12 pg_wal
drwx------ 2 1000 1000 17 May 11 12:27 pg_xact
-rw------- 1 1000 1000 88 Apr 17 2024 postgresql.auto.conf
-rw------- 1 1000 1000 29770 Apr 17 2024 postgresql.conf
-rw------- 1 1000 1000 87 May 22 15:12 postmaster.opts
-rw------- 1 1000 1000 109 May 22 15:12 postmaster.pid
I was just getting silly with globs
Zack
Zack2w ago
Hmmm, but that also seems wrong It should not be in pgdata All of those files you have there hsould be located in /var/lib/postgresql/data in the container
dasunsrule32
dasunsrule32OP2w ago
PGDATA=/var/lib/postgresql/data/pgdata
PGDATA=/var/lib/postgresql/data/pgdata
Zack
Zack2w ago
Oh, you've overwritten the data location 0.o
dasunsrule32
dasunsrule32OP2w ago
I must've done that awhile ago...
Zack
Zack2w ago
That is not something that we accounted for, hmmm
dasunsrule32
dasunsrule32OP2w ago
I can move it up the tree and cp -rvp Well, just re-mount should work actually PGDATA=/var/lib/postgresql/data
Zack
Zack2w ago
Yea, if you do that, it'll fix your immediate problem
dasunsrule32
dasunsrule32OP2w ago
Let me try...
Zack
Zack2w ago
I'm trying to reproduce this locally so I can fix it
dasunsrule32
dasunsrule32OP2w ago
Hrm, crapping out Adjust this?
volumes:
- ${DB_PATH}:/var/lib/postgresql/data
volumes:
- ${DB_PATH}:/var/lib/postgresql/data
to:
volumes:
- ${DB_PATH}:/var/lib/postgresql
volumes:
- ${DB_PATH}:/var/lib/postgresql
It's not liking this change:
initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data".
initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data".
Zack
Zack2w ago
You should leave your volume as it is But remove the PGDATA env And change DB_PATH to /var/lib/postgresql/data/pgdata
dasunsrule32
dasunsrule32OP2w ago
I can't change DB_PATH, that is a local host mount That is where the data is.
Zack
Zack2w ago
Right, but that's just an env, right? Sorry, what is DB_PATH now? The actual value
dasunsrule32
dasunsrule32OP2w ago
root@apps2:/mnt/db/postgres# ls -la total 18 drwx------ 4 debian debian 4 May 22 11:36 . drwxr-xr-x 8 apps apps 8 Jan 21 13:44 .. drwxr-xr-x 2 root debian 2 May 22 11:36 data drwx------ 21 debian debian 27 May 22 11:34 pgdata it created a new data folder
Zack
Zack2w ago
so change the DB_ENV variable to /mnt/db/postgres/pgdata And remove PG_DATA env from the container Volumes should stay like
volumes:
- ${DB_PATH}:/var/lib/postgresql/data
volumes:
- ${DB_PATH}:/var/lib/postgresql/data
dasunsrule32
dasunsrule32OP2w ago
/mnt/db/postgres/pgdata is on the host that is running docker.
Zack
Zack2w ago
Correct So set the docker compose .env variable DB_PATH to /mnt/db/postgres/pgdata
dasunsrule32
dasunsrule32OP2w ago
I have to find all the postgres vars, it's been awhile since I set this up lol
Zack
Zack2w ago
Instead of /mnt/db/postgres I think I have a solution I am about to PR anyway
dasunsrule32
dasunsrule32OP2w ago
Oh, yeah, makes sense.. Now it's trying to change permissions:
root@apps2:/mnt/db/postgres# docker logs -f postgres
Using SSD storage
chmod: changing permissions of '/var/lib/postgresql/data': Operation not permitted
chmod: changing permissions of '/var/run/postgresql': Operation not permitted
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted
fixing permissions on existing directory /var/lib/postgresql/data ... Using SSD storage
chmod: changing permissions of '/var/lib/postgresql/data': Operation not permitted
chmod: changing permissions of '/var/run/postgresql': Operation not permitted
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted
fixing permissions on existing directory /var/lib/postgresql/data ...
root@apps2:/mnt/db/postgres# docker logs -f postgres
Using SSD storage
chmod: changing permissions of '/var/lib/postgresql/data': Operation not permitted
chmod: changing permissions of '/var/run/postgresql': Operation not permitted
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted
fixing permissions on existing directory /var/lib/postgresql/data ... Using SSD storage
chmod: changing permissions of '/var/lib/postgresql/data': Operation not permitted
chmod: changing permissions of '/var/run/postgresql': Operation not permitted
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted
fixing permissions on existing directory /var/lib/postgresql/data ...
Welp, my day is done... hits head on desk
Zack
Zack2w ago
🤣
dasunsrule32
dasunsrule32OP2w ago
root@apps2:/mnt/db/postgres# rm -r pgdata/ This isn't my day
Zack
Zack2w ago
You ran that? 🤔
dasunsrule32
dasunsrule32OP2w ago
I just got off a long flight and I'm exhausted. I'm pulling backups now. But yeah,...
Zack
Zack2w ago
Ah, not the best time to do things like this 😅 At least you have backups 😉 I'll have a fixed database image up shortly if you want to keep your PGDATA var 😛
dasunsrule32
dasunsrule32OP2w ago
Ok, I'm back Rolled back my dataset... Ugh At least I know my backups work! lol Thank you for your help and patience. 🙂
Zack
Zack2w ago
Have you got it working now then with those changes? 🙂
dasunsrule32
dasunsrule32OP2w ago
Nope, I rolling back those changes now to the new one's. Your newer image was wanting to change the permissions and was failing with permission denied.
Zack
Zack2w ago
Was that just because you had deleted the folder though? So docker created the new folder as root, then it couldn't change perms because it runs as 1000:1000 I think now you've restored the folder it should work 😛
dasunsrule32
dasunsrule32OP2w ago
Let me try... fingers crossed.
dasunsrule32
dasunsrule32OP2w ago
Success
No description
Zack
Zack2w ago
Excellent news! 😛
dasunsrule32
dasunsrule32OP2w ago
Let me try to start immich again... Same issues with immich now But hey, we're on the correct db server now though
immich-server | [Nest] 7 - 05/22/2025, 12:04:34 PM LOG [Microservices:EventRepository] Initialized websocket server
immich-server | [Nest] 7 - 05/22/2025, 12:04:35 PM LOG [Microservices:DatabaseRepository] Reindexing clip_index
immich-server | [Nest] 7 - 05/22/2025, 12:04:35 PM LOG [Microservices:DatabaseRepository] Reindexing face_index
immich-server | [Nest] 7 - 05/22/2025, 12:04:34 PM LOG [Microservices:EventRepository] Initialized websocket server
immich-server | [Nest] 7 - 05/22/2025, 12:04:35 PM LOG [Microservices:DatabaseRepository] Reindexing clip_index
immich-server | [Nest] 7 - 05/22/2025, 12:04:35 PM LOG [Microservices:DatabaseRepository] Reindexing face_index
immich-server | Query failed : {
immich-server | durationMs: 2.8418940000001385,
immich-server | error: PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
immich-server | ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`
immich-server | at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
immich-server | at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
immich-server | at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
immich-server | at Socket.emit (node:events:518:28)
immich-server | at addChunk (node:internal/streams/readable:561:12)
immich-server | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
immich-server | at Readable.push (node:internal/streams/readable:392:5)
immich-server | at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
immich-server | severity_local: 'ERROR',
immich-server | severity: 'ERROR',
immich-server | code: 'XX000',
immich-server | file: 'error.rs',
immich-server | line: '125',
immich-server | routine: 'vectors::error::check_client'
immich-server | },
immich-server | sql: '\n' +
immich-server | ' CREATE INDEX IF NOT EXISTS clip_index ON smart_search\n' +
immich-server | ' USING vectors (embedding vector_cos_ops) WITH (options = $$\n' +
immich-server | ' optimizing.optimizing_threads = 4\n' +
immich-server | ' [indexing.hnsw]\n' +
immich-server | ' m = 16\n' +
immich-server | ' ef_construction = 300\n' +
immich-server | ' $$)',
immich-server | params: []
immich-server | }
immich-server | Query failed : {
immich-server | durationMs: 2.8418940000001385,
immich-server | error: PostgresError: pgvecto.rs: The extension is upgraded so all index files are outdated.
immich-server | ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`
immich-server | at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
immich-server | at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
immich-server | at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
immich-server | at Socket.emit (node:events:518:28)
immich-server | at addChunk (node:internal/streams/readable:561:12)
immich-server | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
immich-server | at Readable.push (node:internal/streams/readable:392:5)
immich-server | at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
immich-server | severity_local: 'ERROR',
immich-server | severity: 'ERROR',
immich-server | code: 'XX000',
immich-server | file: 'error.rs',
immich-server | line: '125',
immich-server | routine: 'vectors::error::check_client'
immich-server | },
immich-server | sql: '\n' +
immich-server | ' CREATE INDEX IF NOT EXISTS clip_index ON smart_search\n' +
immich-server | ' USING vectors (embedding vector_cos_ops) WITH (options = $$\n' +
immich-server | ' optimizing.optimizing_threads = 4\n' +
immich-server | ' [indexing.hnsw]\n' +
immich-server | ' m = 16\n' +
immich-server | ' ef_construction = 300\n' +
immich-server | ' $$)',
immich-server | params: []
immich-server | }
Zack
Zack2w ago
I'm not 100% sure, but I believe if you delete the pg_vectors folder it should be happy, in your pgdata folder I think it's got confused somewhere, so deleting that will force it to reindex Which is fine as the server is about to update to vectorchord anyway
dasunsrule32
dasunsrule32OP2w ago
I will cp it out of that folder. I've had enough "rolling" back today lol
Zack
Zack2w ago
Oh... Remove the DB_VECTOR_EXTENSION=pgvecto.rs env Try that first lol That's forcing it to use pgvecto.rs
dasunsrule32
dasunsrule32OP2w ago
👍
Zack
Zack2w ago
Rather than upgrade to vectorchord
dasunsrule32
dasunsrule32OP2w ago
Yeah, that makes sense. So shiny!
root@apps1:/mnt/db/postgres/pgdata# docker logs -f immich-server
Initializing Immich v1.133.0
Detected CPU Cores: 16
Starting api worker
Starting microservices worker
[Nest] 7 - 05/22/2025, 12:12:31 PM LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 7 - 05/22/2025, 12:12:31 PM LOG [Microservices:DatabaseRepository] Creating VectorChord extension
[Nest] 17 - 05/22/2025, 12:12:31 PM LOG [Api:EventRepository] Initialized websocket server
[Nest] 7 - 05/22/2025, 12:12:31 PM LOG [Microservices:DatabaseRepository] Reindexing clip_index
[Nest] 7 - 05/22/2025, 12:12:31 PM LOG [Microservices:DatabaseRepository] Reindexing face_index
[Nest] 7 - 05/22/2025, 12:12:32 PM LOG [Microservices:DatabaseRepository] Reindexed clip_index
[Nest] 7 - 05/22/2025, 12:12:32 PM LOG [Microservices:DatabaseRepository] Reindexed face_index
[Nest] 7 - 05/22/2025, 12:12:32 PM LOG [Microservices:DatabaseRepository] Dropping pgvecto.rs extension
[Nest] 7 - 05/22/2025, 12:12:32 PM LOG [Microservices:DatabaseRepository] Running migrations, this may take a while
[Nest] 7 - 05/22/2025, 12:12:33 PM LOG [Microservices:DatabaseRepository] Migration "1744991379464-AddNotificationsTable" succeeded
[Nest] 7 - 05/22/2025, 12:12:33 PM LOG [Microservices:DatabaseRepository] Migration "1745244781846-AddUserAvatarColorColumn" succeeded
[Nest] 7 - 05/22/2025, 12:12:33 PM LOG [Microservices:DatabaseRepository] Migration "1745902563899-AddAssetVisibilityColumn" succeeded
[Nest] 7 - 05/22/2025, 12:12:33 PM LOG [Microservices:DatabaseRepository] Migration "1746636476623-DropExtraIndexes" succeeded
[Nest] 7 - 05/22/2025, 12:12:33 PM LOG [Microservices:DatabaseRepository] Migration "1746768490606-AddUserPincode" succeeded
root@apps1:/mnt/db/postgres/pgdata# docker logs -f immich-server
Initializing Immich v1.133.0
Detected CPU Cores: 16
Starting api worker
Starting microservices worker
[Nest] 7 - 05/22/2025, 12:12:31 PM LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 7 - 05/22/2025, 12:12:31 PM LOG [Microservices:DatabaseRepository] Creating VectorChord extension
[Nest] 17 - 05/22/2025, 12:12:31 PM LOG [Api:EventRepository] Initialized websocket server
[Nest] 7 - 05/22/2025, 12:12:31 PM LOG [Microservices:DatabaseRepository] Reindexing clip_index
[Nest] 7 - 05/22/2025, 12:12:31 PM LOG [Microservices:DatabaseRepository] Reindexing face_index
[Nest] 7 - 05/22/2025, 12:12:32 PM LOG [Microservices:DatabaseRepository] Reindexed clip_index
[Nest] 7 - 05/22/2025, 12:12:32 PM LOG [Microservices:DatabaseRepository] Reindexed face_index
[Nest] 7 - 05/22/2025, 12:12:32 PM LOG [Microservices:DatabaseRepository] Dropping pgvecto.rs extension
[Nest] 7 - 05/22/2025, 12:12:32 PM LOG [Microservices:DatabaseRepository] Running migrations, this may take a while
[Nest] 7 - 05/22/2025, 12:12:33 PM LOG [Microservices:DatabaseRepository] Migration "1744991379464-AddNotificationsTable" succeeded
[Nest] 7 - 05/22/2025, 12:12:33 PM LOG [Microservices:DatabaseRepository] Migration "1745244781846-AddUserAvatarColorColumn" succeeded
[Nest] 7 - 05/22/2025, 12:12:33 PM LOG [Microservices:DatabaseRepository] Migration "1745902563899-AddAssetVisibilityColumn" succeeded
[Nest] 7 - 05/22/2025, 12:12:33 PM LOG [Microservices:DatabaseRepository] Migration "1746636476623-DropExtraIndexes" succeeded
[Nest] 7 - 05/22/2025, 12:12:33 PM LOG [Microservices:DatabaseRepository] Migration "1746768490606-AddUserPincode" succeeded
Zack
Zack2w ago
👀 :peepoAwesome:
dasunsrule32
dasunsrule32OP2w ago
It's back up 🙂 Thank you so much. That was a pain Should I change the postgres image now to w/o pgvecto.rs? I guess that doesn't exist, nevermind.. that will happen over time I guess?
Zack
Zack2w ago
Feel free to, we'll be suggesting people make that change in a future update
dasunsrule32
dasunsrule32OP2w ago
So image: ghcr.io/immich-app/postgres:16-vectorchord0.3.0-pgvectors0.3.0 to image: ghcr.io/immich-app/postgres:16-vectorchord0.3.0?
Zack
Zack2w ago
And no worries, thanks for uncovering a bug with our new database image 😉
dasunsrule32
dasunsrule32OP2w ago
I guess you're welcome? lol
Zack
Zack2w ago
Yep
dasunsrule32
dasunsrule32OP2w ago
Ok, testing now...
Zack
Zack2w ago
You weren't the only one experiencing it, a few people had PGDATA set, but they were working around it in bad ways 😅
dasunsrule32
dasunsrule32OP2w ago
Success! I've migrated completely from what I can tell. One last question, about the new DB_STORAGE_TYPE config I'm running on rust, but with ZFS arc and l2arc. Should that be changed to HDD?
Zack
Zack2w ago
Depends, if your database files will live on an SSD mostly then you could leave it on SSD probably The parameters we tune are mostly around read speed and random access speed
dasunsrule32
dasunsrule32OP2w ago
Files live on hdd, but once the cache is warmed, it's fast.
Zack
Zack2w ago
Right, but where's your L2ARC?
dasunsrule32
dasunsrule32OP2w ago
SSD
Zack
Zack2w ago
I'm not going to be able to give you a definitive answer really, it depends how often the database files are promoted to ARC/L2ARC Personally I would always have a SSD only pool, and a HDD pool with or without L2ARC on SSD
dasunsrule32
dasunsrule32OP2w ago
No worries, I just wasn't sure if there was a best practive for that outside of SSD/NVME pools.
Logarithm
Logarithm2w ago
FATAL: role "root" does not exist any ideas?
Zack
Zack2w ago
You should make a new help desk thread if you need support
dasunsrule32
dasunsrule32OP2w ago
Summary: Fix was to upgrade to, obviously matching your postgres version with pgvector version: 1. ghcr.io/immich-app/postgres:16-vectorchord0.3.0-pgvectors0.3.0 2. Make sure data was mounted properly in /var/lib/postgresql/data. 3. Disable pgvecto.rs config: DB_VECTOR_EXTENSION=pgvecto.rs After that, everything else came up and migrated was successful. Database server can be moved to: ghcr.io/immich-app/postgres:16-vectorchord0.3.0 once everything has successfully migrated. Thanks @Zack
Zack
Zack2w ago
All good, have a good rest of your day! 🙂
dasunsrule32
dasunsrule32OP2w ago
You too
bavarianbidi
bavarianbidi4d ago
👋 post here (as i have the same pgsql database version running). current status: moved the database image from tensorchord/pgvecto-rs:pg16-v0.3.0 to ghcr.io/immich-app/postgres:16-vectorchord0.3.0-pgvector0.8.0 which works fine. updated immich from ghcr.io/immich-app/immich-server:v1.132.1 to ghcr.io/immich-app/immich-server:v1.133.1 immich is failed to start all the time. i get following error-messages in the log:
Query failed : {
durationMs: 6.255999999999858,
error: PostgresError: could not access file "$libdir/vectors": No such file or directory
at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
at Socket.emit (node:events:518:28)
at addChunk (node:internal/streams/readable:561:12)
at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
at Readable.push (node:internal/streams/readable:392:5)
at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
severity_local: 'ERROR',
severity: 'ERROR',
code: '58P01',
file: 'dfmgr.c',
line: '208',
routine: 'internal_load_library'
},
sql: 'SELECT indexdef, indexname FROM pg_indexes WHERE indexname = ANY(ARRAY[$1, $2])',
params: [ 'clip_index', 'face_index' ]
}
Query failed : {
durationMs: 6.255999999999858,
error: PostgresError: could not access file "$libdir/vectors": No such file or directory
at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
at Socket.emit (node:events:518:28)
at addChunk (node:internal/streams/readable:561:12)
at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
at Readable.push (node:internal/streams/readable:392:5)
at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
severity_local: 'ERROR',
severity: 'ERROR',
code: '58P01',
file: 'dfmgr.c',
line: '208',
routine: 'internal_load_library'
},
sql: 'SELECT indexdef, indexname FROM pg_indexes WHERE indexname = ANY(ARRAY[$1, $2])',
params: [ 'clip_index', 'face_index' ]
}
[Nest] 7 - 05/28/2025, 3:01:47 PM WARN [Microservices:DatabaseService] Could not run vector reindexing checks. If the extension was updated, please restart the Postgres instance. If you are upgrading directly from a version below 1.107.2, please upgrade to 1.107.2 first.
PostgresError: could not access file "$libdir/vectors": No such file or directory
at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
at Socket.emit (node:events:518:28)
at addChunk (node:internal/streams/readable:561:12)
at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
at Readable.push (node:internal/streams/readable:392:5)
at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
severity_local: 'ERROR',
severity: 'ERROR',
code: '58P01',
file: 'dfmgr.c',
line: '208',
routine: 'internal_load_library'
}
[Nest] 7 - 05/28/2025, 3:01:47 PM WARN [Microservices:DatabaseService] Could not run vector reindexing checks. If the extension was updated, please restart the Postgres instance. If you are upgrading directly from a version below 1.107.2, please upgrade to 1.107.2 first.
PostgresError: could not access file "$libdir/vectors": No such file or directory
at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
at Socket.emit (node:events:518:28)
at addChunk (node:internal/streams/readable:561:12)
at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
at Readable.push (node:internal/streams/readable:392:5)
at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
severity_local: 'ERROR',
severity: 'ERROR',
code: '58P01',
file: 'dfmgr.c',
line: '208',
routine: 'internal_load_library'
}
microservices worker error: PostgresError: could not access file "$libdir/vectors": No such file or directory, stack: PostgresError: could not access file "$libdir/vectors": No such file or directory
at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
at Socket.emit (node:events:518:28)
at addChunk (node:internal/streams/readable:561:12)
at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
at Readable.push (node:internal/streams/readable:392:5)
at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
microservices worker exited with code 1
Killing api process
microservices worker error: PostgresError: could not access file "$libdir/vectors": No such file or directory, stack: PostgresError: could not access file "$libdir/vectors": No such file or directory
at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
at Socket.emit (node:events:518:28)
at addChunk (node:internal/streams/readable:561:12)
at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
at Readable.push (node:internal/streams/readable:392:5)
at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
microservices worker exited with code 1
Killing api process
ah, i guess i've found the issue i have to use the image-tag 16-vectorchord0.4.1-pgvector0.8.0-pgvectors0.2.1 which i not have to use as immich will then fail with microservices worker error: Error: The VectorChord extension version is 0.4.1, but Immich only supports >=0.3 <0.4. 😅 ok, using the correct container-image solve all the problems. used 16-vectorchord0.3.0-pgvector0.8.0-pgvectors0.2.1 image tag and immich 0.133.1` is up and running
Zack
Zack4d ago
👍

Did you find this page helpful?