Server unavailable
After updating the immich server, removing old versions with system prune -a and updating the ubuntu server, I started having problems with the server. It keeps changing from available to unavailable all the time. I can even upload photos, but soon after the server appears as unavailable.
53 Replies
:wave: Hey @PPS110,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJAChecklist
I have...
1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
2. :blue_square: read applicable release notes.
3. :blue_square: reviewed the FAQs for known issues.
4. :blue_square: reviewed Github for known issues.
5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like
fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.GitHub
immich-app immich · Discussions
Explore the GitHub Discussions forum for immich-app immich. Discuss code, ask questions & collaborate with the developer community.
FAQ | Immich
User
GitHub
Issues · immich-app/immich
High performance self-hosted photo and video management solution. - Issues · immich-app/immich
docker ps -a
what is the uptime of the containers?updated today
Are you using a reverse proxy? How do you connect to immich
No.
And the rest of the question?
So the immich-server container is restarting, what does
docker compose logs tell you
Also please try to copy/paste the terminal text if you canhere
no logs at all huh
immich_redis | 1:M 26 Jan 2025 10:16:40.966 # Server initialized
immich_redis | 1:M 26 Jan 2025 10:16:40.966 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
What about
docker compose logs immich-serverhere
apart from the restarting not much to see here 👀
If you do
docker compose logs immich-server --follow does anything interesting pop up when the container restarts?here
Sorry, --follow leaves the logs open and shows new ones, is there really no error appearing?
ah found it, sorry
Database does not seem happy
There might be more useful logs in the logs directory of the postgres location, i.e. DB_DATA_LOCATION
How do I see this?
It's quite literally a folder, in the location you're currently at.
-> ./postgres/logs
Can I see anything interesting through pgadmin?
maybe but I have no idea how
GNU nano 7.2 postgresql-2025-01-26_190356.log
2025-01-26 19:03:56.633 UTC [1] LOG: starting PostgreSQL 14.10 (Debian 14.10-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit 2025-01-26 19:03:56.633 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 2025-01-26 19:03:56.633 UTC [1] LOG: listening on IPv6 address "::", port 5432 2025-01-26 19:03:56.647 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2025-01-26 19:03:56.674 UTC [28] LOG: database system was shut down at 2025-01-26 19:03:50 UTC [2025-01-26T19:03:56Z INFO service::utils::clean] Find directory "pg_vectors/indexes/264552". [2025-01-26T19:03:56Z INFO service::utils::clean] Find directory "pg_vectors/indexes/30973". [2025-01-26T19:03:56Z INFO service::utils::clean] Find directory "pg_vectors/indexes/30973/segments/aeda38d5-7123-47e2-8c89-b7e64820ca7f". [2025-01-26T19:03:56Z INFO service::utils::clean] Find directory "pg_vectors/indexes/264552/segments/10baed3b-b230-40a2-b819-d00843efb945". 2025-01-26 19:03:56.716 UTC [1] LOG: database system is ready to accept connections 2025-01-26 19:06:26.095 UTC [1] LOG: received fast shutdown request 2025-01-26 19:06:26.106 UTC [1] LOG: aborting any active transactions 2025-01-26 19:06:26.106 UTC [69] FATAL: terminating connection due to administrator command 2025-01-26 19:06:26.106 UTC [71] FATAL: terminating connection due to administrator command 2025-01-26 19:06:26.109 UTC [1] LOG: background worker "logical replication launcher" (PID 41) exited with exit code 1 2025-01-26 19:06:26.114 UTC [36] LOG: shutting down 2025-01-26 19:06:26.196 UTC [1] LOG: database system is shut down
2025-01-26 19:03:56.633 UTC [1] LOG: starting PostgreSQL 14.10 (Debian 14.10-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit 2025-01-26 19:03:56.633 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 2025-01-26 19:03:56.633 UTC [1] LOG: listening on IPv6 address "::", port 5432 2025-01-26 19:03:56.647 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2025-01-26 19:03:56.674 UTC [28] LOG: database system was shut down at 2025-01-26 19:03:50 UTC [2025-01-26T19:03:56Z INFO service::utils::clean] Find directory "pg_vectors/indexes/264552". [2025-01-26T19:03:56Z INFO service::utils::clean] Find directory "pg_vectors/indexes/30973". [2025-01-26T19:03:56Z INFO service::utils::clean] Find directory "pg_vectors/indexes/30973/segments/aeda38d5-7123-47e2-8c89-b7e64820ca7f". [2025-01-26T19:03:56Z INFO service::utils::clean] Find directory "pg_vectors/indexes/264552/segments/10baed3b-b230-40a2-b819-d00843efb945". 2025-01-26 19:03:56.716 UTC [1] LOG: database system is ready to accept connections 2025-01-26 19:06:26.095 UTC [1] LOG: received fast shutdown request 2025-01-26 19:06:26.106 UTC [1] LOG: aborting any active transactions 2025-01-26 19:06:26.106 UTC [69] FATAL: terminating connection due to administrator command 2025-01-26 19:06:26.106 UTC [71] FATAL: terminating connection due to administrator command 2025-01-26 19:06:26.109 UTC [1] LOG: background worker "logical replication launcher" (PID 41) exited with exit code 1 2025-01-26 19:06:26.114 UTC [36] LOG: shutting down 2025-01-26 19:06:26.196 UTC [1] LOG: database system is shut down
How much memory do you have?
To me this looks like postgres is taking too long to start due to lack of memory and is then being shut down by the healthcheck
8gb
Status
running
HA State
none
Node
pps
CPU usage
0.00% of 8 CPU(s)
Memory usage
57.76% (4.62 GiB of 8.00 GiB)
Bootdisk size
35.00 GiB
Hm that should be plenty, assuming there's not a lot of other stuff going on
Can you share your compose file?
I don’t think you will get ready to accept connections if it’s a resource issue
They aren't, right?
Oh wait the timestamps are confusing
Look like the DB was up for 3 minutes
Could be a liveliness probe but it’s showing healthy above
WARNING: Make sure to use the docker-compose.yml of the current release:
https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
The compose file on main may not be compatible with the latest release.
#
name: immich
services:
immich-server:
container_name: immich_server
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
# extends:
# file: hwaccel.transcoding.yml
# service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
volumes:
# Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
- ${UPLOAD_LOCATION}:/usr/src/app/upload
- /etc/localtime:/etc/localtime:ro
env_file:
- .env
ports:
- '2283:2283'
depends_on:
- redis
- database
restart: always
healthcheck:
disable: false
immich-machine-learning:
container_name: immich_machine_learning
# For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
# Example tag: ${IMMICH_VERSION:-release}-cuda
image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
# extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
# file: hwaccel.ml.yml
# service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the
-wsl version for WSL2 where applicable
volumes:
- model-cache:/cache
env_file:
- .env
restart: always
healthcheck:
disable: false
redis:
container_name: immich_redis
image: docker.io/redis:6.2-alpine@sha256:905c4ee67b8e0aa955331960d2aa745781e6bd89afc44a8584bfd13bc890f0ae
healthcheck:
test: redis-cli ping exit 1
restart: always
database:
container_name: immich_postgres
image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
environment:
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_USER: ${DB_USERNAME}
POSTGRES_DB: ${DB_DATABASE_NAME}
POSTGRES_INITDB_ARGS: '--data-checksums'
volumes:
# Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
- ${DB_DATA_LOCATION}:/var/lib/postgresql/data
healthcheck:
test: >-
pg_isready --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" exit 1;
Chksum="$$(psql --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" --tuples-only --no-align
--command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')";
echo "checksum failure count is $$Chksum";
[ "$$Chksum" = '0' ] || exit 1
interval: 5m
start_interval: 30s
start_period: 5m
command: >-
postgres
-c shared_preload_libraries=vectors.so
-c 'search_path="$$user", public, vectors'
-c logging_collector=on
-c max_wal_size=2GB
-c shared_buffers=512MB
-c wal_compression=on
restart: always
volumes:
model-cache:
I changed my .env to version 1.124.2 and rebuilt the server. It seems to have worked.
I tried to return to the latest version and the problem returned. I kept it on 1.124.2.Did the problem only happen when you access the timeline on the web?
no. at app it happens too
at app never goes online.
What Alex means is, after a restart and it working, does it break as soon as you open the timeline web, or before?
So as long as you avoid
/photos, will it stay running?I think I might have spot the issue. Probably related to the memory query
It usually goes offline. Pressing F5 makes it go online. Clicking on any photo returns it offline. Pressing F5 a few times makes it go online and opens the photo. If you exit the photo it goes offline again.
Can you go to
http://your-ip:2283/user-settings?isOpen=feature+memories
and disable it
then restart the server and enter the home pagebut I have to play to version 1.125.3, right?
Any version that you reported the issue above is fine
I assume 1.125.3?
continued with the same error.
Hmm
How long have you had this instance and how many assets are there?
Whenever I disable the feature it comes back
Did you hit save?
after some testing, I must have made my server in October last year. I have 18 thousand photos and 2900 videos
Sorry. I didn't notice the save button at the bottom of the screen. It looks like it worked.
Okay, so a bug on our end. I wonder that trigger the error
Let’s temporarily disable that feature until we fix the bug
Congratulations! I hope you return soon. This is a really cool feature.
Can you help access the database and check the assets table to see if the
fileCreatedAt column or the localDateTime column has any weirdness?You will have to be a little more specific in the instructions.
from the command line run
then run
and then run
can you post the output here?
Those looks normal
hmm
I'll be out for 10 minutes. If you need anything else, please leave it here and I'll get back to you with an answer.
I am back
We will need to investigate this further
thank you for helping out