Update to 1.53 completly failed: Immich doesn't start.
Hi everyone.
I' tried to update Immich to V1.53 and it doesn't start. The last versión i have running I think was 1.49. As a clue, it was the last version using altran repositories and bertmelis tensorflow for non AVX CPU.
I'm running Immich installed using Portainer on a Debian Server with Openmediavault software for NAS. The computer is an old Intel Atom 330 with 2 Gb RAM (+ 8 Gb Swap)
Before update, I tested the new configuration on a Virtual Machine running debian 11 on an i5 7200U host. The test was OK.
I copied and updated the config (docker compose and stack.env) into the NAS but Immich didn't start and I got an Error 500: connect ECONNREFUSED. I check the event log in Portainer and the problem seems to be related with Typesense and Machine-Learning. Do Typesense and Machine Learning support non-AVX CPU?. I'd like to test immich with this features disalbles. How can I setup this? Best Regards Alberto P.S. Sorry for any mistake using my poor English. I'm from Spain.
I copied and updated the config (docker compose and stack.env) into the NAS but Immich didn't start and I got an Error 500: connect ECONNREFUSED. I check the event log in Portainer and the problem seems to be related with Typesense and Machine-Learning. Do Typesense and Machine Learning support non-AVX CPU?. I'd like to test immich with this features disalbles. How can I setup this? Best Regards Alberto P.S. Sorry for any mistake using my poor English. I'm from Spain.
102 Replies
Ml should support non avx now, but typesense doesn't
Thank you very much. I disable Typesense following the instructions on this post (https://github.com/immich-app/immich/issues/2224) and immich started fine.
GitHub
[BUG] connect ECONNREFUSED 172.18.0.8:3001 due to typesense · Issue...
The bug Can't connect to immich after update to version 1.53.0. I have tried this using the default docker-compose.yml and .env using an empty media folder and still recieve the same result. I ...
Hi again.
Something went wrong removing Typesense. Now, immich-server can start, but it goes down every little time. I was looking the logs and the trouble seems to be with Machine-Learning.:
root@trantor:~# docker logs immich_machine_learning /bin/sh: 0: cannot open ./entrypoint.sh: No such file /bin/sh: 0: cannot open ./entrypoint.sh: No such file /bin/sh: 0: cannot open ./entrypoint.sh: No such file /bin/sh: 0: cannot open ./entrypoint.sh: No such file /bin/sh: 0: cannot open ./entrypoint.sh: No such file /bin/sh: 0: cannot open ./entrypoint.sh: No such file /bin/sh: 0: cannot open ./entrypoint.sh: No such file /bin/sh: 0: cannot open ./entrypoint.sh: No such file /bin/sh: 0: cannot open ./entrypoint.sh: No such file /bin/sh: 0: cannot open ./entrypoint.sh: No such file /bin/sh: 0: cannot open ./entrypoint.sh: No such file /bin/sh: 0: cannot open ./entrypoint.sh: No such file [....] /bin/sh: 0: cannot open ./entrypoint.sh: No such file root@trantor:~#Do I need to disable anything more? Best Regards Alberto
Your docker-compose file is out of date can you double check with the latest version?
Hi Alex,
I 've compared my stack file (I'm using portainer) with the docker-compose.yml i've just downloaded from github and I didn't noticed any differences except the comment lines for disabling Typesense and some old (and maybe useless) lines from my old stack file like NODE_ENV:Production. I upload my stack file. Please, would you check it? Perhaps I didn't noticed something important.
Best regards.
the entry point of machine learning section is incorrect
There is no entry_point for the new file 🙂
Hi. Thank you very much. It seems to run ok:
root@trantor:~# docker logs immich_machine_learning
[2023-04-13 18:40:43 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2023-04-13 18:40:43 +0000] [1] [INFO] Listening at: http://0.0.0.0:3003 (1)
[2023-04-13 18:40:43 +0000] [1] [INFO] Using worker: gthread
[2023-04-13 18:40:43 +0000] [13] [INFO] Booting worker with pid: 13
[2023-04-13 18:40:43 +0000] [14] [INFO] Booting worker with pid: 14
I've downloaded the docker-compose file about one hour ago from this link: https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml and the immich-machine-learning section was this:
immich-microservices:
container_name: immich_microservices
image: ghcr.io/immich-app/immich-server:release
entrypoint: ["/bin/sh", "./start-microservices.sh"]
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
env_file:
- .env
depends_on:
- redis
- database
- typesense
restart: always
Please, can you check the file from the link. I think the file is outdated. The link is in https://immich.app/docs/install/docker-compose. I use this link because is easier to find than in github.
Best regards
Alberto
weird, typesense was working on previous version
I have a non-avx cpu
after upgrading to 1.54 it's not working for me anymore and typesense won't start
Typesense has never worked on non-avx CPU AFAIK
I don't know why it worked on mine before then
I was able to search
You can still search with metadata
using
m:<metadata>I was able to search for items
like, "cookie"
hmm
strange
ikr
Did you change anything?
no
Cna you post your .env?
1 sec
You have
TYPESENSE_ENABLED=false
so it was never enable beforeno, I added that in
before it wasn't there
I'll try and downgrade and see if I can get it working
yup downgraded to v1.53 and immich works with typesense
lolwut
did the log different for typesense?
what is typesense log on 1.54 and what on 1.53
I don't know typesense isn't logged
is the only thing I have to do change the logging driver?
when you say didn't work, is there a specific container doesn't start?
err I get that error something can't connect port
e refused
let me find it again
ECONNREFUSED
on which container?
Can you capture a screenshot?
I don't know which container fails though
let me grab logs
for logs v1.53:
for logs v1.54:
and then 503 forever
but I guess this should be a separate ticket
it's just the same error
isn't it so weird on my non-avx cpu typesense works on 1.53 but not 1.54?
in 1.53 you can also see
yes but it eventually starts
This is typesense indexing and the server waiting it to be completed
1.54 it doesn't start after waiting
correct
Can you enable typesense log on 1.54?
yes do I just comment out
?
Yes correct
1 sec
typesense version was pinned so I am not sure what is happening here
connect ECONNREFUSED 172.28.0.7:3001
Hmm
ok now do this
bring down all containers
remove typesense container
remove typesense volume
then bring up everything again
on 1.54 or 1.53?
1.54
err what command would this be?
docker container rm immich_typesense
root@snuminipc:~/immich# docker container rm immich_immich_typesense
Error response from daemon: No such container: immich_immich_typesense
root@snuminipc:~/immich# docker container rm immich_typesense
Error response from daemon: No such container: immich_typesense
compose file:
container_name: immich_typesensedocker ps pleaseerr it's not running right now
do I start it and then ps?
you can do
docker container lsnot listed there either
does docker compose down remove the container too?
yeah probably
can you now do
docker volume ls?local immich_model-cache
local immich_pgdata
I deleted the tsdata one
ok starting it up again now
ok
now bring everything back up
works
wtf
I don't have avx
😂
damnit typesense wtf
So probably issue with indexing causing the server fails to communicate with Typesense?
I have no idea - I thought non-avx isn't supported
why is it working here? N5095 CPU
and why did tsdata need to be deleted? What's in there?
typesense indexes data
oh I can't search now
let me reindex
deleted it causes typesense to reindex
oh crashed
err
I tried to search for car
do I need to reindex something from jobs?
No it should reindex itself
CLIP job is just for generated the embedded array and added to the asset
oh god now I get a bunch of Write to disk failed errors
will restore old document
immich_typesense | E20230423 04:35:39.356161 196 collection.cpp:364] Write to disk failed. Will restore old document
immich-immich-microservices-1 | [Nest] 1 - 04/23/2023, 4:35:39 AM LOG [SearchService] Indexing 64742 assets
but looks like to be expected
I've never seen that error message before
Internal server error - 500 - Internal Server Error
undefined
should I delete tsdata and try again?
Maybe
I just don't understand why it works on 1.53 and not 1.54
ok deleted tsdata
neither do I
for typesense you should search for phase like "red car parking"
I don't need to run ENCODE CLIP
Run machine learning to generate clip embeddings ?
You don't need to run any job
ok
if you did that before
ok now it fails
only last few lines are relevant I think
immich_machine_learning | 172.31.0.7 - - [23/Apr/2023:04:42:11 +0000] "POST /sentence-transformer/encode-text HTTP/1.1" 200 10380 "-" "axios/0.26.1"
immich-immich-server-1 | [Nest] 1 - 04/23/2023, 4:42:11 AM ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not Found
immich-immich-server-1 | ObjectNotFound: Request failed with HTTP code 404 | Server said: Not Found
missing CLIP embeddings?
I clicked missing and it doesn't run anything
yeah it just crashes now if I try to search for anything
I don't know why it was working on 1.53
:8113jailpepe:
;_; maybe someone will provide instructions to compile for non-avx
what system do you use that has avx?
You can run Typesense from a different machine that has AVX and just point the server to that typesense container
any modern CPU should have AVX
yes - but what do you use? do you have a NAS?
N5095 /is/ recent >_>
No I have a dedicated server
it just doesn't have avx reee
Ryzen 3700
o
🥲
maybe with the storaxa, cwwk NAS, or lenovo NAS I can finally get something for self-hosting with avx
in a synology form factor
You can run a free oracle VPS
and run typesense off that machine
LOL
yeah it should run just fine there
or downgrade to 1.53 ;_;
let me see if I can get it working again
I promised myself I'd never use the oracle vps for anything important because of the horror stories
people's accounts getting cancelled etc
0 customer support
yada yada
Typesense is stateless though so it is not really important 😄
You can move typesense around and it just reindexing
or just disable typesense entirely
yep but then you will miss out a lot of feature
maybe bertmelis will come around with a no-avx docker image
like future facial recognition
ugh
in the future I might upgrade this minipc anyway
to a proper NAS
actually on 1.53 it looks like I can't create new clip encodings anyway? but somehow was able to create them before
idk
janky
GitHub
[BUG] 500 getaddrinfo ENOTFOUND immich-server · Issue #2055 · immic...
The bug Can't access Immich web page since version v1.51.0. I get 500 getaddrinfo ENOTFOUND immich-server error. _immich_server_logs.txt The OS that Immich Server is running on Debian 11 Versio...
wheee gunna build my own image
I got it working
non-avx users can run the build command from https://github.com/typesense/typesense#build-from-source
GitHub
GitHub - typesense/typesense: Open Source alternative to Algolia an...
Open Source alternative to Algolia and an Easier-to-Use alternative to ElasticSearch ⚡ 🔍 ✨ Fast, typo tolerant, in-memory fuzzy Search Engine for building delightful search experiences - GitHub - t...
reference that built image and typesense will be working
😄
Nice!