Postgress DB turned faulty?
After updating to 139.4 my next backup failed. The issue is in the pg_dumpall step. Does this mean I need to restore from backup or can this be salvaged?
Pure chance this happened after the backup or is anyone facing the same issue?
docker exec immich_postgres sh -c \ 'pg_dumpall --clean --if-exists --username=postgres > /tmp/immich.sql'
pg_dump: WARNING: page verification failed, calculated checksum 51456 but expected 24139 pg_dump: error: Dumping the contents of table "smart_search" failed: PQgetResult() failed. pg_dump: error: Error message from server: ERROR: invalid page in block 1104 of relation base/16384/17309 pg_dump: error: The command was: COPY public.smart_search ("assetId", embedding) TO stdout; pg_dumpall: error: pg_dump failed on database "immich", exiting
23 Replies
:wave: Hey @Hinfty,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
2. :blue_square: read applicable release notes.
3. :blue_square: reviewed the FAQs for known issues.
4. :blue_square: reviewed Github for known issues.
5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.Docker logs during relevant time
immich_redis | 1:M 27 Aug 2025 06:32:01.010 * 100 changes in 300 seconds. Saving... immich_redis | 1:M 27 Aug 2025 06:32:01.011 * Background saving started by pid 87 immich_redis | 87:C 27 Aug 2025 06:32:01.087 * DB saved on disk immich_redis | 87:C 27 Aug 2025 06:32:01.088 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB immich_redis | 1:M 27 Aug 2025 06:32:01.112 * Background saving terminated with success
Also smart search returns zero results regardless of query and generate thumbnails shows 14 failed. Rerunning smart search completes after a second, which seems weird
Yes, the db is corrupted
So drop and restore backup?
yeah
When restoring the db from backup, do I need to wipe the upload location too?
I restored without wiping and still see the thumbnail generation problems and the server starts cycling
Also I'm getting error 500 now
šØ Error - Something went wrong
Error: 500 ()
Stacktrace
Error: Error: 500
at Object.ot [as ok] (http://192.168.178.54:2283/_app/immutable/chunks/CC_8Wp91.js:1:5407)
at async Promise.all (index 1)
at async c (http://192.168.178.54:2283/_app/immutable/chunks/CSUBkz_5.js:1:524)
at async o (http://192.168.178.54:2283/_app/immutable/chunks/DiW5tmNF.js:1:177)
at async rt (http://192.168.178.54:2283/_app/immutable/nodes/0.C9w5uMjb.js:1:2081)
at async Fe (http://192.168.178.54:2283/_app/immutable/chunks/CT584u9k.js:1:16238)
You can't restore without wiping
Shit alright. It's just annoying to restore tens of gigs from storage just to reset by a few days.
Alright, I'll try with more time on my hands. Thanks
Uh, I think you misread
You should NOT wipe the upload location. the database does not contain those files
@Hinfty beep
oops I did
So I can keep the files that are not yet listed in the old DB backup?
Unfortunately you will need to re-upload the missing ones
No issue.
But that means they are already in the upload location and will be orphaned
Because there is no DB reference to their ID, I'm assuming
Correct
There used to be an easier way to locate these files but theat is currently not functional
So I've reset the DB to several previous backups, but found that once I connect my phone and sync, the immich_server container restarts a lot. I see the 500 error in the web. I didn't see anything in the docker logs. Is there an option to identify the reason for a container restart?
Did you upgrade after the restore?
Apart from docker logs there aren't any
I didn't downgrade for the restore, but I'm not sure that's an issue?
So the issue right now seems to be the immich_server cycling once I start uploading from my mobile. Using docker logs, the server just cuts out without any error msg. Health checks and restart now deactivated, but still nothing in logs or dmesg.
Any more ideas how to see what makes the container cut out?
What hardware is this?
This is a raspberrypi with PiOS
Please add
VIPS_NOVECTOR=1 to the .env file and compose up again šGood job, that was the problem. I really had a problem with a corrupt db, but when I saw the broken/missing thumbs, I thought this was linked to a bad DB in the backup, so went back further.
I hadn't read the issue on GIT, because it was about external data, which I am not using. But it seems Raspi's need this to work.
Thanks a lot for helping me out on this one!
This thread has been closed. To re-open, use the button below.