Redis used all my disk space, how to handle it?
Hello all, I was running a smart search job, when I noticed my server was stuck for a while with no CPU usage. I tried restarting and such, only to check some logs and find that Redis failed, because... it has used up my entire boot drive (290GB as it's only needed for boot).
Either way, I have Immich's database on separate (larger) drives, so it's only the boot drive affected. I just mainly want to know how to handle the full boot drive?
Should the Redis data be deleted, should it be moved elsewhere? Can Redis be pointed to store data on a separate drive?
Sorry if this may seem a little dumb lol. Thanks for any help :)
22 Replies
:wave: Hey @Chribs,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
2. :blue_square: read applicable release notes.
3. :blue_square: reviewed the FAQs for known issues.
4. :blue_square: reviewed Github for known issues.
5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.I doubt redis is talking up this much space
It's much more likely you have a bunch of old unused docker images
What's the output of
docker system df ?I've removed some unnecessary files, but i can't find what's taking up the space. I'll troubleshoot it later, as it may be something else on the server. I though it was Redis because the drive only became full after it was running smart search, nothing else was done between then. I've managed to recover 20GB for now, so theres that.
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 6 6 4.013GB 74.78MB (1%)
Containers 6 5 24.46kB 0B (0%)
Local Volumes 6 3 4.365GB 30B (0%)
Build Cache 0 0 0B 0B
Yeah not much
But like I said in my other message, I'll look into this a bit later, as it may be something else.
For now though, can you point Redis at a different location rather than the boot drive? Just to avoid any future issues, if not it's fine
Sure but there’s no evidence that is the actual issue
You should start to research what is actually taking up the space then report back if it’s related to immich in any way
I'm currently doing so
There's barely any space being used by docker
No, just found the issue
Unless you changed redis away from the default and set it to use a mounted folder, which I really doubt, any redis storage would be included there
There is another folder there which was taking majority space. Just hold one one sec, I'm checking things
Okay I removed the folder, it was taking about 56GB, not sure how the disk got full. I can't remember where it was at beforehand, but I'm sure it wasn't that full. Either way, removed the folder and Smart Search is running again with no errors (so far). I'll monitor this closely and update if needed.
Sorry for any hassles, thanks for the help
(OUTDATED, see below update)
It has once again done this.
I've been tearing my hair our the entire morning trying to resolve this
So i re-ran smart search and it was going fine until about 6am this morning (3 hours ago from now) when i noticed it failed again, for the same reason. Checked and sure enough, the entire drive was filled.
So here I am again. Now the drive had about ~70GB free when i started yesterday again. (Nothing was added, as I was sleeping. Unless i somehow added 70gb while sleepwalking)I am not sure how much data Redis uses, but surely it can't use this much?
Still trying to figure this out, but will post further info soon ,as I'm still going through everything. Please let me know, if I should provide any specific info here
My current issue is finding what exactly is taking up all the space, as the biggest things on the root drive are as follows:
13G /home
12G /var
7.6G /usr
Yeah, so either I'm an idiot whos missed something very obvious, or... I dont even know anymore
What are the outputs of
df -h, lsblk, and mount?Filesystem Size Used Avail Use% Mounted on
udev 7.4G 0 7.4G 0% /dev
tmpfs 1.5G 1.5M 1.5G 1% /run
/dev/sdb1 292G 288G 0 100% /
tmpfs 7.5G 0 7.5G 0% /dev/shm
tmpfs 5.0M 12K 5.0M 1% /run/lock
/dev/sda 880G 384G 451G 46% /mnt/database
/dev/sdc1 458G 199G 236G 46% /mnt/media
tmpfs 1.5G 84K 1.5G 1% /run/user/1000
-
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 894.3G 0 disk /mnt/database
sdb 8:16 0 298.1G 0 disk
├─sdb1 8:17 0 297.1G 0 part /
├─sdb2 8:18 0 1K 0 part
└─sdb5 8:21 0 976M 0 part [SWAP]
sdc 8:32 0 465.8G 0 disk
└─sdc1 8:33 0 465.8G 0 part /mnt/media
sdd 8:48 0 953.9G 0 disk
└─sdd1 8:49 0 953.9G 0 part /media/root/84ee8b8d-42f1-4641-bebc-8af52ef7899c
-
(will send in a moment)
-
and as said above with du on everything
838G /mnt
384G /media
13G /home
12G /var
7.6G /usr
1.6G /root
686M /opt
153M /boot
12M /etc
1.6M /run
76K /tmp
16K /lost+found
4.0K /srv
and docker df:
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 6 2 4.013GB 3.684GB (91%)
Containers 2 1 0B 0B
Local Volumes 8 1 4.365GB 4.365GB (99%)
Build Cache 0 0 0B 0B
Sorry if I'm somehow missing something here
What does
du -hd1 /mnt say?255G /mnt/backup
199G /mnt/media
384G /mnt/database
838G /mnt
Each one mounted is a separate drive
/mnt/backup doesn't have a disk mounted on it
So anything written there is going into your root fs
Which is why it's filling up
the what
One sec, let me check
Oh my god
Okay, remounted it correctly and cleared the files. Root drive has space again.
I have no clue how this was missed :/
I'm so sorry I brought this chaos here, thank you so much for pointing that out