upgrade to v1.33.0 keeps filling up my db storage
hello everyone, I am running immich in an RKE2 cluster. I just upgraded to v1.33.0 and when the server is clustering, it keeps filling up my PV.
before the upgrade: my postgres_db PV was set to 4GB, it filled up few times, so I increased the storage to 8GB, then to 10GB, then 12GB. and it filled up again.
Can someone please verify if this is expected from the new database image, if yes, how much storage should I expect the new database image to consume?
Thanks!
29 Replies
:wave: Hey @sandex,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.Pastebin
Using SSD storagePostgreSQL Database directory appears to contain a...
Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
Pastebin
Initializing Immich v1.133.0Detected CPU Cores: 16Starting api work...
Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:
attached db and server logs in the pastebin ^
@mertalev did say that the migration will take up a good amount of space. I'm not sure if there are specific numbers...?
oh yeah? i could not find anything on the release notes, could you please point me to where @mertalev said so? i just want to make sure i did not fudge up something while upgrading
Oh that was a private channel 😅
The indexing mode we use is optimized for performance and memory usage at the cost of more disk space. There's another mode that uses less disk space, but there's a significant performance cost involved
Advanced Features | VectorChord
Scalable Vector Search in Postgres
Also, the way the migration is done ends up using more disk space as well. I don't have exact numbers, but I guess you could say that the vectors get stored 5 times at peak: once for the base data, once for changing the data type, once for changing it again, once for indexing this data with the new index, and once because the old index isn't truly dropped until everything is done
oh i see. i read the doc that you linked and it seems that it would need about 80GB for 5M vectors which is a LOT of storage space I do not have. although it seems from my log, I have 130k vectors which might not take as much space. however, i will have to rollback to older version till i figure out my storage situation. I do think it would have been helpful to include an 80GB increase in storage in the release notes so users could take an informed decision before going through an upgrade. Thank you for your help!
To be clear, the index doesn't use anywhere near that at your scale. They just say that because the index can scale to millions of vectors, but if you have 130k it'd be maybe 2GB for the search index and another 2GB for the face index, assuming 1 face per asset
I'd guess that 20GB would be enough, though the earlier attempts would still take up space in your DB until they get vacuumed up. You can manually vacuum the DB to help free up more space
And you would need less after the migration completes
yes. exactly what i was about to ask. i am suspecting the multiple container crashed filled up the storage. can you please point me to how i can cleanup the older records(vacuum?) and start fresh?
You can ssh into the database container, run
psql -U postgres immich and run these commands:
This will maximize your chances of having enough disk spacethanks, will try and let you know! the last command goes as is? with the brackets?
Yup
so my disk was initially 4 GB before the upgrade. i increased to 14 GB and ran the vacuum command. the disk still shows 12GB used and 2 GB available.
Is anything else besides you connected to this database?
no its a container
the server container is down. i have exec'd into the db container
Hmm, I wonder what's taking that space up. Is it
pg_wal? Did you run VACUUM FULL or just VACUUM?i copied the commands as is 🙂
Try running
select pgvectors_upgrade(); and restarting the containeri have increased the db storage to 15 gb after vacuum and now the server seems to be running some migrations. will check this if it fails again.
the server is now up and my db has used about 13 GB storage from the initial 4. this is what is using storage:
7.6G base
582M core
392K global
2.0M log
4.0K pg_commit_ts
4.0K pg_dynshmem
8.0K pg_hba.conf
4.0K pg_ident.conf
16K pg_logical
44K pg_multixact
4.0K pg_notify
4.0K pg_replslot
4.0K pg_serial
4.0K pg_snapshots
4.0K pg_stat
100K pg_stat_tmp
112K pg_subtrans
4.0K pg_tblspc
4.0K pg_twophase
20K pg_vectors
4.0K PG_VERSION
5.1G pg_wal
2.3M pg_xact
4.0K postgresql.auto.conf
32K postgresql.conf
4.0K postmaster.opts
4.0K postmaster.pid
not sure if i should run the
select pgvectors_upgrade(); command?You don't need to in this case
Running
VACUUM ANALYZE; now should free some more space as well
One other thing you can do is to run ALTER SYSTEM SET max_wal_size = '1GB'; and restart. Might want to double check that it took effect by running SHOW max_wal_size; after the restart. This will shrink pg_wal at the cost of some performancei guess it is fine for now. especially if it comes with a performance hit. its now using about 90% of my allocated 15 GB. which i am fine with.
I had to increase my VM OS size as well for this update 😅
but maybe it is better to include the "drop index" and "vacuum" command in the release so other people might not have to manually exec into the container (some are probably not proficient enough to do it)?
thanks for your help @mertalev.
No problem! #18440 should help a bit with this