Errors after the latest update
Setup: Docker Swarm on multiple nodes (including a Synology NAS)
36 Replies
Error log from immich_server
Can you post your setup? Also if you bring down and bring up the whole stack again, would it help?
I tried that multiple times, after prune buit that didn't seem to help
Ah it seems to have an issue with migrations
@Zack @jrasm91 do you guys have any pointers?
That's very odd
The database is somehow in an unexpected state, where shared_links doesn't exist where it is expected to
Wait... That's saying the entire shared_links table doesn't exist
@ExtremeInspection have you ever gone into your database and manually changed anything?
Nope. I'm not that adventurous 🙂
Well I'm really not sure how that table could be missing
Do you know how to connect to a database and run queries?
Yes I can do that
If you could connect then run describe on the shared_links table
I believe
\d shared_links is shorthand for that which should work
Make sure you're in the immich database context before running itDid not find any relation named "shared_links".
Yea, that makes sense...
Hmmm
Can you restart the stack?
What version did you just upgrade from?
They've already done that, if TypeORM is trying to apply this migration, it's because it believes all previous ones have been applied correctly
Maybe try starting only the server container
But somehow the shared links table is missing
Could you run
\d user_token for me?I'm not too sure.. I have an auto-update set up with Shepherd.. I can look up the version with the signatures, if it is important
If you've been keeping up with the versions, it's not that important. I was just wondering if it jumped a bunch of versions at once somehow
Although even that shouldn't cause a problem
immich=# \d user_token
ERROR: cache lookup failed for relation 16410
Uhhh
That's a scary error, lemme look up what that means
Can you shut down the server and microservice instances then try run the command again?
Could you also retrieve the postgres server logs?
I notice you are using NFS for your underlying filesystem. Is this setup with async or sync on fstab?
Just tried... Same error
immich=# \d user_token
ERROR: cache lookup failed for relation 16410
FYI I have about 10 minutes before I need to head off
This one is the most important question
nfs:
driver: local
driver_opts:
type: "nfs4"
o: addr=10.20.11.5,rw,nolock
device: ":/volume1/Docker-Data/immich"
Have you had any issues with the server hosting the NFS share going down, or the connection between the servers being unstable?
It looks like the connection is over your local LAN so probably only the former is relevant
Double checked on the server side.. It is set to async
Ah
Async makes things faster but is much more prone to corruption or data loss
Due to it not finishing writing to the actual server before responding saying everything is done
Basically, what I think has happened is some kind of data corruption. I'm not sure how else the database could be in this state. Our migration software believes it has created this table and is on a later script. postgres stores tables in seperate files, so I believe it's possible that the migration table wasn't affected but the shared_links table has been deleted in the underlying storage
I'm not sure how else to diagnose this really. My suggestion would be to start fresh and re-import your assets (and probably don't run postgres over NFS)
However, if you did want to try and repair your current setup, you could do so by manually running the scripts to create the shared links table
I can try that route
Oh god
Yes your database is fucked 🤣
That log is full of permanent write errors
😄
Alright.. rebuilding the db then
Good to know my educated guess was bang on 🤣
So I would suggest spinning up a new immich cluster, and using the CLI to import all the files from the old storage folder
And probably disable the async option for NFS
Or even better, don't use NFS for your database storage xD
Well.. I'm running a docker swarm with storage on a NAS for HA... Let me see how best to handle this...
For import via CLI is there a CLI for immich that you are referring to here?
Yea, there is a cli tool with a docker image for running it
There is documentation on the docs site
I think there may be an issue with it since the latest version,but it may have been fixed already
Yea, that bug is already fixed. Anyway I have to head to bed, good luck with the re-build!
Thank you. Let me spend some time to go through that. Thanks for the help though!!