PostgreSQL Keeps Restarting and server failed
The issue I encountered is that after running normally for some time, PostgreSQL stops functioning properly, which causes the server to fail. Below are my deployment environment and detailed problem description.
Deployment Environment:
Host 1: x86 Synology NAS, CPU J3160, 8GB RAM
Deployment: Initialized Immich1, added Synology photo folder as an external library but did not start photo processing tasks. The server was seperated into server and microservice, and the microservice container was stopped, while other four remained in use.
docker-compose.yml:
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
- /volume4/homes/dafei/Photos:/mnt/media/Photos:ro #external album
- /etc/localtime:/etc/localtime:ro
Host 2: x86 Ubuntu (ESXi VM), CPU N5105, 8GB RAM
Deployment: Immich2 was set up primarily to run the microservice. All five were in use.
Used NFS to mount the following directories:
Synology photo folder /volume4/homes/dafei/Photos (mounted to /mnt/DS920/photos).
Synology Docker's Immich folder /volume4/docker/immich (mounted to /mnt/DS920/immich).
docker-compose.yml:
container_name: immich_server
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
- /mnt/DS920/photos:/mnt/media/Photos:ro
- /etc/localtime:/etc/localtime:ro
.env:
UPLOAD_LOCATION=/mnt/DS920/photos
DB_DATA_LOCATION=/mnt/DS920/immich/postgres
---
Initially, the program ran smoothly, but over time, the external library statistics became inaccurate and failed to update.
Upon reviewing logs, database connection errors were observed, with Host 1's PostgreSQL service reporting issues.
Logs from Host 1's PostgreSQL:
The following repeated messages were logged:
PostgreSQL Database directory appears to contain a database; Skipping initialization
2024-11-28 01:33:07.696 UTC [1] LOG: redirecting log output to logging collector process
2024-11-28 01:33:07.696 UTC [1] HINT: Future log output will appear in directory "log".
13 Replies
:wave: Hey @flywalt,
Thanks for reaching out to us. Please follow the recommended actions below; this will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker compose ps docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
Checklist
1. :ballot_box_with_check: I have verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: I have read applicable release notes.
3. :ballot_box_with_check: I have reviewed the FAQs for known issues.
4. :ballot_box_with_check: I have reviewed Github for known issues.
5. :ballot_box_with_check: I have tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: I have uploaded the relevant logs, docker compose, and .env files, making sure to use code formatting.
7. :blue_square: I have tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
If this ticket can be closed you can use the /close command, and re-open it later if needed.
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:You should never use NFS for postgres, or any database
You should fix that first and then see if the issue corrects itself
Also, am i understanding that youre running two full copies of immich (4 or 5 containers each) sharing the same data directories..?
Yes, my photo library is stored on the NAS, but its performance is insufficient to efficiently handle photo processing tasks (a folder with 200,000 images). After referencing this link: https://immich.app/docs/administration/jobs-workers, I came up with the idea of splitting tasks between two Immich instances. I plan to use Immich1 on the NAS for primary browsing and data storage, while the other host, with a relatively more powerful CPU, assists in photo processing. To achieve this, I used NFS for mounting.
ok, that's definitely not how you implement that though
If you want to use a second system, you run only the microservices container, and connect it to the postgres and redis servers using env vars
not all 5
you should probably start over completely with a new install
by doing so, should I still use nfs to connect to mount postgres on nas? And should I use docker compose to start the whole setup and shut down other four after that OR just use docker to setup microservices only? There isn't a clear guide in the doc so I'm a bit confused:monkaW:
no, you cannot use NFS for postgres, at all
you cannot use the whole setup. ONLY the microservices container
then what is the proper way to connect to postgres on another host?
using the environment vars, DB_HOSTNAME, etc
you wil need to mount the UPLOAD_LOCATION only using NFS
Your second PC does not seem that much stronger, are you sure this is worth the time/hassle?
Haha, maybe you are right, but my NAS is indeed TOO weak. I feel the main issue lies in generating thumbnails and video previews. It pushed my NAS to run at 99% CPU usage for three days and still didn’t finish even 1/10 of the task. However, Synology’s built-in PHOTO app completed the same task in about three days without consuming so many resources. That’s why I came up with the idea of splitting the workload.
makes sense, you could setup the second microservices as described then
I have another question: if I only start the microservices on Host 2, does that mean I need to manually stop the microservices on Host 1? Also, does Host 1 need to modify any configurations to connect to the microservices container on Host 2 and assign tasks to it? Apologies if the question is a bit complicated.
both of the microservices containers can be running, and no, they pick up the jobs from Redis
Thx! I'll try it now