Files not being deleted despite > 30 days
Example file UUID:
0dc60527-2d1d-451c-89ba-106be84e3d45
visiting at https://immich.domain.tld/photos/0dc60527-2d1d-451c-89ba-106be84e3d45 shows the proper photos
The original is able to be downloaded
The filename in the sidebar does match the path on disk: upload/library/usernamehere/IMAGE/2022/04_partner_images_Apr/2022-04-16_T221817_IMG_5978__0dc60527-2d1d-451c-89ba-106be84e3d45.PNG
Template runs with no errors:
/admin/repair page shows no offline or untracked files
Storage template: {{filetypefull}}/{{y}}/{{#if album}}{{{album}}}{{else}}{{MM}}_{{MMM}}{{/if}}/{{y}}-{{MM}}-{{dd}}_T{{HH}}{{mm}}{{ss}}_{{{filename}}}__{{assetIdShort}}60 Replies
:wave: Hey @Zeus,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
2. :blue_square: read applicable release notes.
3. :blue_square: reviewed the FAQs for known issues.
4. :blue_square: reviewed Github for known issues.
5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.Can you check the moves table?
oooh wait they might be deleted š
š
wait why is the trash not emptying though?
deletedAt | 2024-03-28 15:18:48.264227-0430 days
At least by default
yes.... that's about 370 days š
Oh, 2024
Woops
I overlooked the year, sorry
you're good.
Other nightly jobs are running?
E.g. memory stuff?
yep memories look good
does the trashing command look for > 30 days or specifically 30? is it possible it got missed and now it's too old to find?
We only filter on deletedAT
<=
Just to verify; not a ro external lib?
no, all internal
Can you try manually deleting one?
Ideally keeping an eye on logs
it worked, no issues
gone from disk as well
And it's also gone from the fs now?
Hmmm
pre deletion DB, isOffline is false
this is the only log I got:
[Nest] 7 - 04/02/2025, 5:13:11 PM LOG [Microservices:TrashService] Queued 1 asset(s) for deletion from the trash
That's the one that worked successfully then, right?
Can you verify if some of your assets get cleared from trash?
Or is it just not clearing at all?
that's the one that worked just now when I manually did it
I think the trash overall is working fine.. it's mostly stuff I recognize from < 30 days
can I mock the query to select the items? where does trashedBefore come from?
We
which is then used in the
query
Ah there you go lol
trashedDays is the database configselect * from assets where "deletedAt" <= '2024-12-31' and "isOffline" = 'f'; returns the 8 rows of interest
so I think we might have an actual bug hereHmmm
Let's walk through this https://github.com/immich-app/immich/blob/main/server/src/services/asset.service.ts
So we get those assets back in 175
We push them all to the chunk array
We call queueChunk() in 183
should trashedDays be in the json from
system_metadata.system-config ?We queue asset deletion for all those assets with
deleteOnDisk: true -- this part we do all over the place, especially also when manually triggering it
I don't recall exactly, if it's the default I think we don't save it
We always fill up with defaultsgotcha
ok not the issue then, I don't have that json key in the DB
And the query itself is also fine š
There isn't this much more that can be wrong lol
yep, query returns all files expected for deletion..
and the deletion dates do sound correct. it's not at all recent
so it's not like it got re uploaded from the phone or something
Yea
I'm assuming there aren't any helpful logs from tonight?
I can check in the morning. I check the logs pretty often and haven't seen anything. I don't think I have last night since it just upgraded to 1.131
I sort of doubt it given the manual delete worked fine
The fact that the trash generally is clearing though is killing me š
Which logging level do you have set atm?
default, through journald
I can increase the level for server and microservices
Maybe just for tonight if that'd be fine for you
I really don't have any idea right now š
So weird
yeah i'll do verbose and see
wow that's a lot of logs lmao
Yeah it really goes lol
Alternatively... are you willing to run a dev instance against your prod db? š
Iād just clone the database, which is fine
But Iād need to clone the assets as well ?
Only the relevant ones
Was thinking about adding the asset cleanup job to that "job create" endpoint
So that you can trigger that whenever you want to
I keep forgetting... when was the status column supposed to be set to
trashed or deleted and what did it mean if an asset is active but has a non-null deletedAtTo more easily test
I mean, we don't look at that column at all afaict
Yeah I could just restore into a new database pretty easily
We have
.where('status', '=', AssetStatus.TRASHED) in the query no?
And the JSON above has activeThis is the one that's called when you click the empty trash button
We queue this https://github.com/immich-app/immich/blob/main/server/src/services/asset.service.ts#L188 in the nightly job https://github.com/immich-app/immich/blob/main/server/src/services/job.service.ts#L223
Unless I am completely lost right now
the files that are in the trash from more recently are indeed status = trashed..
Yeah generally that should absolutely be the case, Mert is right with that
all 8 old rows do have status = active
But I don't think we care about that for the nightly cleanup, which is wrong
logs very bland, just a lot of face matching jobs after this
wait... they got deleted lmao
..........
I can't do this anymore :monakS:
I don't understand wtf changed lol
unless the first file that I manually deleted was somehow blocking the rest??
You didn't change the status field by any chance?
no I didn't touch the DB other than the queries
Ok I really have no idea then š
But it works! :peepoAwesome:
š¤·āāļø