Uploaded images, deleted, emptied trash. File persists in file structure and database still
In the past two or three days, I've noticed several files that I've uploaded, deleted and emptied trash still exist in the folder/file structure on the server. If I try to re-upload any of these files, I get an error that states the file exists. I'm not seeing any errors in the logs that seem to indicate any issues with the files. I have also tried removing and recreating the docker containers using the same compose file and still have the same issue. Not sure what to check next.
Running on Truenas Scale Electric Eel with custom yaml configuration, behind NPM reverse proxy and authentik MFA. This instance has been up and running without issues for several months.
20 Replies
:wave: Hey @octotron15,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:attaching log files
please share the asset ID of one of the files
/usr/src/app/upload/library/eb1eadbb-1e8a-4e04-a760-1455e47d9063/2025/2025-01-04/IMG_5092.jpeg
/trash/d2825417-99f8-4f65-9805-80ccf548881b
Ok so its in the trash?
Good question. This image along with several other images and videos have been deleted and trash emptied. The trash for the user is empty, but the file still appears in the /trash/guid in the address bar or the actual file in the server file directory
is this related to the external library at all? or these are all mobile/web uploads
have you confirmed that the docker host system has permissions to delete files from the mountpoint?
These images are all in the mobile and web upload section. I don’t have any issues with deleting images in an external library
have you given it 48+ hours to let the deletes after trash be synced to disk?
as in 48 hours after emptying the trash
I have confirmed that the docker host and containers have root access to all of these files, however, I have noticed that if an image is uploaded via the web, the file permissions are root:root -rwxr—r— I’m not sure if the files should have should be rwxrwxrwx for uploads.
I have not given it 48hrs for the files to clear after emptying the trash. Is this documented somewhere that it takes 48hrs for deleted files to sync back to the actual disk?
no, 744 is fine
no, but it takes time for the cron / system jobs to take effect
so give it a day or two and check back
Ok, that makes sense. Is there any way to run the system jobs to force/test it or do I simply have to wait for 1-2 days?
I don't think so
Got it. I’ll put this on hold for a bit and check to see if the files clear out after a day or two and can re-upload them.
Thanks!
Good morning. I am thinking there must still be an issue. Its now been almost 2 days and the original file still exists in the hard drives on the NAS as well as shows up in the trash on the database, but is not visible in the actual trash for the user. I'm also showing several images in the front end that show a broken thumbnail and updates to dates are not persisting with reboots or re-organizing files. Finally, I'm seeing that the jobs for thumbnails generation are stuck at 3/3740 and have been stuck there over the weekend. I've attached a fresh set of logs as well.
This looks like your installation is relatively broken in multiple ways. I would question/investigate the reliability of your underlying NAS connection
I see a lot of ffmpeg EOFs / corrupt file errors
Yeah, it definitely seems like something isn't working properly. To expand on how my configuration is setup: single Truenas system, docker containers all run on an NVME SSD (immich server, redis, postgres, worker), and files are stored on a raidz1 4-drive spinning rust. No other issues with any other services or containers, only immich seems to be impacted. To confirm, do all files used by immich and its dependencies need 744 permissions? If all of that looks good, what is my best option at this point? Should I look at rebuilding my immich instance from scratch and re-importing all files, setting up users and restoring shares, or is there anyway to fix any corruption or figure out what is broken and fix that? I'm relativley new on how immich is set up, so I'm not quite sure what the best course of action would be in this case.
where is the database stored?
I haven't really seen this set of errors before tbh
the database itself is stored on the NVME ssd. For example: path/to/store/immich/db, /immich/server /immich/redis /immich/machine-learning
As a final follow up to this, I dug into more of the errors and looked at the files based on the GUID's and found that several files it was choking on came from an external folder with corrupted/invalid files (files that were labeled as images, but were in fact only 1kb shortcuts. I cleared all of these corrupted files, re-deployed immich using the latest .env and .yml configs and then let it re-process all of the files. I also found that although I had renamed the container name immich-redis, and immich-postgres, there is another redis container on my stack that was also labeled redis. I found that I cannot change the service name for immich as it appears to be hard-coded to look for service "redis" and "database" (can we change this?), so I changed the rest of the container names and services and no longer have conflicting redis issues. So far things have been cleared up nicely and I haven't seen any errors in any of the logs, so I think we're good to go ahead and close this out. Thanks for your help in pointing more closely to the log files. I know a bit more what to look for if there is another issue.