Immich and dragonfly
I have the strange problem when i update the dragonfly cluster a new pod is master. Because of that there is an ip change and immich wants to use the old ip address. I can only fix that with a restart of the immich pods.
25 Replies
:wave: Hey @DrAg0n141,
Thanks for reaching out to us. Please follow the recommended actions below; this will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker compose ps docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
Checklist
1. :ballot_box_with_check: I have verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: I have read applicable release notes.
3. :ballot_box_with_check: I have reviewed the FAQs for known issues.
4. :ballot_box_with_check: I have reviewed Github for known issues.
5. :ballot_box_with_check: I have tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: I have uploaded the relevant logs, docker compose, and .env files, making sure to use code formatting.
7. :ballot_box_with_check: I have tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
If this ticket can be closed you can use the /close command, and re-open it later if needed.
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:Okay, what exactly is it that you want from us?
Other than "use a hostname, not an IP" there's not much help we can offer
I am using the hostname, i think that immich does resolve the hostname only one time. and when the ip has changed it uses the cached ip of the hostname.
what do you mean, "immich wants to use"? the app or the server?
The server, the app does not use redis or dragonfly
ok. the server doesn't really have any control/opinion as to what IP it's on
this would be an issue with the pod migration / clustering service
immich listens on all available IPs by default
It's the other way around @Zeus
he's using an external redis cluster, dragonfly
Yes the problem is from the immich server to the redis cluster (dragonfly)
oh I see interesting
so the REDIS_HOSTNAME is not re resolving?
It is resolving one time, when the ip has changed it does not get the new address.
It is, but the IP doesn't re-resolve on update
Not really an immich issue 👀 but still interesting
I think your best bet would be using an intermediate proxy
I am running this all in an kubernetes cluster, normally i have a load balancer service in front. That does not help.
I meant immich -> proxy for redis -> redis
It can probably even run in the cluster
But immich won't re-resolve the redis IP, and your redis cluster isn't able to get a fixed IP, so the only solution would be to place something that can do both in between
What proxy can i use for that?
Something dead simple like NPM?
Yes that can work.
But only for interest why does immich not re-resolve the ip?
It might, eventually, how long have you waited? 👀
Over a day 😅 it does not fixed it by itself
If Immich is pointed at a kubernetes Service then afaik that service should have a stable IP even if the backing endpoints change and so Immich shouldn't need to do anything
I thought so too, but every dragonfly upgrade immich crashes.
Can you post the logs? I don't know if it's at all related, but BullMQ requires a specific configuration for Dragonfly: https://www.dragonflydb.io/docs/integrations/bullmq
BullMQ | Dragonfly
BullMQ
https://github.com/drag0n141/home-ops/blob/master/kubernetes/main/apps/database/dragonfly/cluster/cluster.yaml
Thats my config, both is set
Any movement on getting Immich to not cache the IP (or renew cache on write failure)? Immich still needs a forced restarted after every Dragaonfly update or leader change as it then tries to write to a read-only replica.
I have ArgoCD, PaperlessNGX, Authelia using Dragonfly with no issues after leader change.
I haven't investigated but that'd probably be something to take up with the bullmq library
That said, maybe just don't bother fixing it as we're looking to drop redis entirely
FYI opened issue: https://github.com/dragonflydb/dragonfly/issues/4972