immich microservices containers keep restrting
hcr.io/immich-app/immich-server:release "/bin/sh ./start-mic…" 36 hours ago Restarting (1) 28 seconds ago immich_microservices
164 Replies
Can you bring down the whole stack and bring it backup
the monitor the log to see if anything abnormal?
yea one sec
lets try
omg
i forgot my portainer password
so it took me 1.5 hours to figure out how to change it
or get it back
i dont see any abnormalities
oh wait
ther are
I do have 2 ymls btw
not sure why
hey alex
have you taken a look at these?
yeah
can you bring down the stack, remove the microservice container
then recreate the stack
the library we used for reverse geocoding sometimes has issue fetching the updated geo info
ok
thanks
do i recreate the stack through docker compose?
Docker Compose [Recommended] | Immich
Docker Compose is the recommended method to run Immich in production. Below are the steps to deploy Immich with Docker Compose.
here are my logs
Can you walk me through what did you do?
OK
sorry
ok
I shut down all docker containers
docker rm immich-microservices
docker start $(docker ps -a -q)
thats it
oh wait
i did docker compose up -d
now do this
docker compose down
this is what i did
then
docker prune
then docker compose upok
within the immich folder right/
ahh ok
i see waht you did
makes sense
lets try
correct
docker system prune
sorry
docker container pruneoih fuck
HOLY FUCK
did it remove your database volume?
im not usr
docker volume ls
still intact
no worry there
ok
phew
i was scared
now do
docker container pruneim doing this all in the immich folder
so i think it only affects the docker compose yml under that folder right?
not the docker composes in the other folders?
not necessary
docker command is system wide
im going to cry
fml
but system prune usually just remove dangling stuffs
gotcha
stuffs that are being used aren't touched
these are the containers i have left
so i think im ok
yep
ok
now do
docker compose uplets do docker compuse up
ok
a bunch of stuff is happenin
yeah just wait a bit
okk
thank you
how long wll this go for?
what are you seeing now?
oh that is just normal log
if you don't see anything else it should work fine now
oh ok
lets check the logs
i think all is good
We are golden
Thank you
No problem
Nice BMW btw
Hehe thanks!
Friendly reminder to backup your database 😁
Oh dang.
How can I do that?
Or what is the best way to do that?
This is all it takes for a db backup
docker exec -t immich_postgres pg_dumpall -c -U postgres | gzip > "/volume1/docker/immich/dump.sql.gz"
gunzip < ~/Downloads/dump.sql.gz | docker exec -i immich_postgres psql -U postgres -d immichthank you so much!
btw alex, its doing it again 😭
i just updated
i dont see any issues in any of the containers
im going to do the entire process that you told me to do over again
immich_server | Request #1682231971126: Sleeping for 4s and then retrying request...
interesting
ok so the job list method worked
You need to replace the path to the dump SQL with a path that exists on you machine
im so dumb
lmaooo
thakn you
ill make a folder within the immich directory
once i execute the command, does it take a long time to complete?
No it was pretty quick for me
i hope i did it right
Looks right. Maybe depending on the hardware it could take a bit longer
How big is your library?
about 10K photos so far
but they are all live photos. i dont know how to turn them off lmao
Oh really
I don't think you terminated the quote for your path lol
Hit Ctrl+c and try it again
lmaoo
lets try
If you have an open quote that's missing a closing one, enter goes to a new line
oh ya
ocker exec -t immich_postgres pg_dumpall -c -U postgres | gzip > "/home/concac/immich/sqldump/dump.sql.gz"
i think it worked
😄
Yes
Looks good
yay
thanks!!!
this keeps on pausing on its own
beucase microservice crashed
damn
crashed again
with the error you showed in the other thread
the geocoding/csv parser in that library has lots of problem
same error
oh wow
it keeps on restarting
HMM
You did remove and it still restarting?
yea
let me try again
docker stop immich_microservices
docker container prune
docker compose up -d
looks right?
no
RIP
docker compose down
oh
then docker rm immich_microservices
then docker compose up
done deal
❤️
cam ong
không có gì!
ok so docker compose removed the entire stac3k
i just brougt it up again
[Nest] 1 - 04/29/2023, 1:57:29 AM WARN [MetadataExtractionProcessor] Reverse geocoding is enabled
ok
its good now I believe
yeaa i think its good now
thank you
:6308cowroll:
holy smokes
we got errors
😭
pretty sure this is IO bottle neck
pause tag object and encode clip
i really dont know how to check the IO
i have two 10 TB drives
maybe its mergerfs?
but i doubt it cos i use the same server for my plex media server
after the thumbnail generation is done, clear tab tag object and encode clip pipeline, then just run the missing one on tag objects and encode clip
the IO bottle neck is different than storage
done deal
this is pretty much resource handling, Immich need to schedule these job better to not hogging and crashing things
i see. i set up grafana just for this
gotcha
i gave it so much cores to
and ram
so pretty much not your fault, we have some work to do to optimize
well that is different from disk IO
im sorry
im a noob
HDD is always slower than SDD in term of read and write
yea most def
its gonna be expensive to get SSDs for immich ehhehe
so when the write speed exceed the maxmimum of the HDD, all of the other operation will basically come to a halt
should i puse this too
interesting
yeah pause that too
hmm
alrighty
thank you
You can just do docker stop immich_microservices and docker rm immich_microservices
❤️
later on clear the queue and only run missing one
gotcha. will pin this
thank you
Or even better yet
docker-compose up --force-recreate immich-microservices
thank you both!
i will stop taking up your time on this fine friday night
i will continue to back up everything
no problem man, drinking beers and writing code
😄
we def have some work to do in term of resource usage
but that is the optimization story for later on
:armleft: :immich: :armright:
Thank you for everything
The thing is, you don't need SSDs for everything. The originals can go on slow storage, but if thumbnails and encoded-videos are one ssd or nvmes the whole app would be super snappy. The only jobs that need to read the originals are extract exif and generate the jpeg thumbnail, the rest use the jpeg thumbnail which, if on an ssd, would be read super fast. There's a thread on github about mounting thumbs and encoded videos on a different drive than library.
ahhh
i will have to look into that
what would be a good size SSD/NVME for thumbnails alone?
thank you heaps
is this supposed to take a while to finish?
no movement for about an hour
logs look good so far
hmm the timing is off
You mean the timezone or something else?
The thumbnail job in particular is a bit annoying. There's two thumbnails per asset a jpeg one and a webp one. First the jpeg one is made and then the webp one is added to the queue (at the end). So it takes one off and puts another one on. Because of that I think the queue stays pretty much the same size until you are 50% of the way done.
I apologize for not being specific. Yes the timezone is off. I'll have to check my machine. Will that affect any immich processes?
oh wow. 2 thumbnails per image. Gotcha.
So I'll just let immich_microservices do its thing and hope it'll finish sometime soon haha
thank you again for you response
yea it looks like immich_microservices is chuggin away
There are a few situations when timezone of the local machine is used I think... not 100% sure what the impact of changing it would be.
local container*
I think if the image doesn't have any timezone information in the metadata it might fallback to local container timezone, etc.
I mean, if there's gps data or other tags it'll use that to figure out the timezone, but assuming it doesn't have any of that I think it might use the timezone of the container.
I think it is UTC+0 by default, which is what you're seeing. I'd probably just leave it as is.
I'm currently at the "Encode Clip" step on the jobs tab
this is the error that im getting
Looks like some timeouts. Is you computer maxed on CPU or disk read/write?
I’m not sure. I gave the VM 24 cores
I don’t think it should be maxed
I’ll check when I get home
some errors while doing storage template migration
Hmm there seem to be a lot of issue with your instance, what did you do that cause it?
what operations have been done so I can try to reproduce it?
no idea!
😭
ive logged everything ive done in this ticket
hehe
latest error
just removed all containers
pruned
and then brought everything back up again
doing some jobs 😭
Error: ffprobe exited with code 1
ffprobe version 4.4.1 Copyright (c) 2007-2021 the FFmpeg developers
built with gcc 10.3.1 (Alpine 10.3.1_git20210424) 20210424
configuration: --prefix=/usr --enable-avresample --enable-avfilter --enable-gnutls --enable-gpl --enable-libass --enable-libmp3lame --enable-libvorbis --enable-libvpx --enable-libxvid --enable-libx264 --enable-libx265 --enable-libtheora --enable-libv4l2 --enable-libdav1d --enable-postproc --enable-pic --enable-pthreads --enable-shared --enable-libxcb --enable-libsrt --enable-libssh --enable-libvidstab --disable-stripping --disable-static --disable-librtmp --enable-vaapi --enable-vdpau --enable-libopus --enable-vulkan --enable-libsoxr --enable-libwebp --enable-libaom --disable-debug
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
upload/upload/7876bfc4-da79-41f5-90db-861a60a1eeb8/468b3ce0-8f4b-4c4a-88d5-b830ac838aee.mov: No such file or directory
at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
at ChildProcess.emit (node:events:527:28)
at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)
Hello, I'm not a daily node.js developer, but I can spend some time digging that
CSV_RECORD_INCONSISTENT_FIELD_LENGTHS bug.
But firstly, may I know if is someone on it or if some work has been done yet?
https://github.com/immich-app/immich/issues/1963#issuecomment-1536696901GitHub
[BUG]: Corrupted Reverse Geocoding CSV File · Issue #1963 · immich...
tomayac/local-reverse-geocoder#63 Possible solutions: Fix upstream Detect the issue, delete the csv file/directory, and try again
I think there is a fix upstream but it cause some issue with the build for Armv7. So we are planning to drop support for armv7 to solve this issue and a few more other issues. So you don’t need to work on that
@Alex Ok, thank you! I have already prepared PR in the upstream that should bring back the armv7 support. But we will se if it will be accepted.
https://github.com/tomayac/local-reverse-geocoder/pull/69
GitHub
Changed max-old-space-size to just below 4GB - Bring back 32bit arc...
Dirty fix for #67
Disclaimer: I'm not a Node.js developer. I'm just facing the problem of #67 and deduced that this would be a fast solution for the problem.
Ah that was you
☺️
immich micro services container started restarting on its own again
i had to bring down the containers and prune and bring them back up again
nonow i hav this error
removed the immich containers just now
brought them back up.
fixed it so far
more errors 😭
I am considering dropping support for armv7 and update that geo coding library to solve this issue
🙏
thank you !
❤️
https://github.com/immich-app/immich/pull/2396 there is fix that doesn't require dropping armv7