Stack won't restart in portainer - noob
Was getting the update notification on my iOS app, so figured I'd update my stack. I have done this once before and all went well, this time not so much. Went into portainer, went into the stack, then editor, scrolled down and hit update stack. Then hit the re-pull and deploy. I let it run for awhile and it hasn't come back. Some of the containers are running, some are exited, some say unhealthy. I followed a guide on how to set this all up and it's pretty overwhelming, but it's been working great for the past couple of months, so I'm a little confused now.
Any ideas? Is there any way to basically "start over" without losing all of my data? I assumed that's kinda what happens when you update these containers anyways.
19 Replies
The first step when containers aren't starting (or are misbehaving in some other way) is to check their logs for errors
That'll usually tell you what's wrong
I went into events, as I don't see a logs section (I'm sure it's somewhere), here is the last 10
2023-09-27 07:53:40 container Container Immich-MICROSERVICES exited with status code 1
2023-09-27 07:53:31 container Exec instance exited
2023-09-27 07:53:31 container Exec instance exited
2023-09-27 07:53:31 container Exec instance started: /bin/sh -c wget --no-verbose --tries=1 --spider http://localhost:3000/ exit 1
2023-09-27 07:53:31 container Exec instance started: /bin/sh -c curl -Lk "${HEALTHCHECK_URL}" exit 1
2023-09-27 07:53:31 container Exec instance created: /bin/sh -c wget --no-verbose --tries=1 --spider http://localhost:3000/ exit 1
2023-09-27 07:53:31 container Exec instance created: /bin/sh -c curl -Lk "${HEALTHCHECK_URL}" exit 1
2023-09-27 07:53:30 network Container disconnected from immich_default network
2023-09-27 07:53:29 container Exec instance exited
2023-09-27 07:53:29 container Exec instance started: /bin/sh -c curl -f http://localhost:8080/ || exit 1
Though, like my post states, total noob. I think I was looking more for if my update process is wrong and what the process should be like...
You can try stopping the stack and then re-deploying it again.
It should take a bit to shut everything down. Then the green start this stack will show up again.
I'll give that a go
So I've tried starting and stopping. repull and deploy.
I just read something about opening ports for typesense, that doesn't seem to have changed anything
PROXY and WEB both say unhealthy.
Are those the only ones unhealthy? Try restarting just those containers
They kinda hang there and then DB exits itself
just restarted those containers and same thing
Can you get the actual container logs? They should have errors in them that point to the problem
WEB:
[hooks.server.ts]:handleError connect ECONNREFUSED 172.27.0.2:3001
[hooks.server.ts]:handleError connect ECONNREFUSED 172.27.0.2:3001
[hooks.server.ts]:handleError connect ECONNREFUSED 172.27.0.2:3001
[hooks.server.ts]:handleError connect ECONNREFUSED 172.27.0.2:3001
[hooks.server.ts]:handleError connect ECONNREFUSED 172.27.0.2:3001
[hooks.server.ts]:handleError getaddrinfo ENOTFOUND immich-server
[hooks.server.ts]:handleError connect ECONNREFUSED 172.27.0.2:3001
[hooks.server.ts]:handleError connect ECONNREFUSED 172.27.0.2:3001
[hooks.server.ts]:handleError getaddrinfo ENOTFOUND immich-server
[hooks.server.ts]:handleError getaddrinfo ENOTFOUND immich-server
lot of that
[hooks.server.ts]:handle connect ECONNREFUSED 172.27.0.6:3001
[hooks.server.ts] Unable to get my user {
message: 'connect ECONNREFUSED 172.27.0.6:3001',
code: 'ECONNREFUSED',
stack: 'Error: connect ECONNREFUSED 172.27.0.6:3001\n' +
' at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)'
}
[hooks.server.ts]:handleError connect ECONNREFUSED 172.27.0.6:3001
[hooks.server.ts]:handle connect ECONNREFUSED 172.27.0.6:3001
[hooks.server.ts] Unable to get my user {
message: 'connect ECONNREFUSED 172.27.0.6:3001',
code: 'ECONNREFUSED',
stack: 'Error: connect ECONNREFUSED 172.27.0.6:3001\n' +
' at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)'
}
[hooks.server.ts]:handleError connect ECONNREFUSED 172.27.0.6:3001
[hooks.server.ts]:handleError getaddrinfo ENOTFOUND immich-server
Listening on 0.0.0.0:3000
[hooks.server.ts]:handleError connect ECONNREFUSED 172.27.0.2:3001
[hooks.server.ts]:handleError connect ECONNREFUSED 172.27.0.2:3001
[hooks.server.ts]:handleError connect ECONNREFUSED 172.27.0.2:3001
DB giving me this over and over
2023-09-29 14:43:31.740 PDT [1] FATAL: database files are incompatible with server
2023-09-29 14:43:31.740 PDT [1] DETAIL: The data directory was initialized by PostgreSQL version 15, which is not compatible with this version 16.0 (Debian 16.0-1.pgdg120+1).
chmod: changing permissions of '/var/run/postgresql': Operation not permitted
PostgreSQL Database directory appears to contain a database; Skipping initialization
2023-09-29 14:43:54.153 PDT [1] FATAL: database files are incompatible with server
2023-09-29 14:43:54.153 PDT [1] DETAIL: The data directory was initialized by PostgreSQL version 15, which is not compatible with this version 16.0 (Debian 16.0-1.pgdg120+1).
Your postgres image version is wrong. You probably don't have a specific tag set (which you really should!) and then updated the image
ahhhhh, any suggestions on a tag
You can't switch between major version like that.
Just pin it to version 15
I'm a noob, but would postgres:15 be fine
I've only dabbled enough to use tags like latest or beta
You can lookup tags on docker hub or GitHub container registry
alright, I'll do some digging and give that a go, thanks
both are still unhealthy and DB exited
wait, I think I mistyped something while running that, brb
hot damn, you've done it!
@bo0tzz & @jrasm91 , thank you both
And your figured out tags all on your own 😎
I appreciate the nudge 🙂
just wanna let you know that I had the same problem, and changing to postgres:15 fixed it! power to community!
Heck yes. I followed that Marius hosting guide, so I’m sure that’s where some folks are getting messed up