Immich•2mo ago

Enabling Storage Templates ended in jobs not completing

I went ahead and enabled Storage Templates because I wanted a more Synology Photos-esque experience, where the filenames were based off of the date and time that the photo was taken, as well as having the images all stored in easy to access photos in case I ever wanted to copy out the raw jpeg files. After enabling it, I ran the Storage Template migration job. It says "Active 1, Waiting 38." It's been like that for several hours now. While I was waiting for it to do its thing, I also went and added a bunch of names to the faces. Now, facial recognition says "Active 1, Waiting 2,305." I'm not fully sure what to do here, as it just appears to be stuck. The file and folder names are still very wild, and those jobs seem to be unable to progress.

31 Replies

Immich•2mo ago

:wave: Hey @EthanD, Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:. References - Container Logs: docker compose logs docs - Container Status: docker ps -a docs - Reverse Proxy: https://immich.app/docs/administration/reverse-proxy - Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA Checklist I have... 1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time). 2. :ballot_box_with_check: read applicable release notes. 3. :ballot_box_with_check: reviewed the FAQs for known issues. 4. :ballot_box_with_check: reviewed Github for known issues. 5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy). 6. :blue_square: uploaded the relevant information (see below). 7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable (an item can be marked as "complete" by reacting with the appropriate number) Information In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider: - Your docker-compose.yml and .env files. - Logs from all the containers and their status (see above). - All the troubleshooting steps you've tried so far. - Any recent changes you've made to Immich or your system. - Details about your system (both software/OS and hardware). - Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h). - The version of the Immich server, mobile app, and other relevant pieces. - Any other information that you think might be relevant. Please paste files and logs with proper code formatting, and especially avoid blurry screenshots. Without the right information we can't work out what the problem is. Help us help you ;) If this ticket can be closed you can use the /close command, and re-open it later if needed.

EthanDOP•2mo ago

This appears to be an issue that is open on GitHub right now as well

Sergey Katsubo•4w ago

Hey @EthanD To better understand your case/setup it would be helpful to have addition info - as requested in the auto-post above - compose file and .env file - logs - info about your storage (if you have external network/drive mounts and so on) What github issue did you mention, btw?

EthanDOP•4w ago

I'm having a difficult time getting the compose and .env files into here...I've sshed into the NAS, and I've opened the files with Vi, which I think is the only text editor available to me, but I can't seem to copy and paste to my clipboard. Only within Vi itself. Do you have a suggestion on how I can get the text out of Vi so tha tI can provide the requested information?

Sergey Katsubo•4w ago

You can print/output file content instead of editing it. If you run vi FILE then try cat FILE

EthanDOP•4w ago

Here are the docker compose and env files

env.txt

dockercompose.txt

EthanDOP•4w ago

And here is the Github issue I mentioned: https://github.com/immich-app/immich/issues/20436

Immich•4w ago

[Issue] Storage Template Migration with Failed Asset (immich-app/immich#20436)

Sergey Katsubo•4w ago

Thank you, I'll have a look at the github issue. The compose and .env look fine, pretty standard.

I ran the Storage Template migration job. It says "Active 1, Waiting 38."

If you still have pending active job, then we can fetch its details (from Redis), hopefully it gives us info about the problematic asset. This rather long command will fetch all data from the storageTemplateMigration queue in Redis. It will print it to console/terminal. Copy-paste into NAS terminal:

docker exec -i immich_redis bash << 'EOF'
KEYS="immich_bull:storageTemplateMigration:*"
two_lines_to_kv='{ sub(/^[0-9]+\) /, "") ; if (NR % 2) { key = $0 } else { print key ": " $0 } }'
xrange_to_csv='/^[0-9]+-[0-9]+$/ { if (NR > 1) print output ; output = "id=" $0 ; next } { if (!key) { key = $0 } else { output = output ", " key "=" $0 ; key = "" } } END { print output }'
redis-cli --scan --pattern "$KEYS" | sort | while read -r key; do
  type=$(redis-cli type "$key")
  echo -e "\n==== $key ($type) ===="  
  case $type in
    "string") redis-cli get "$key" ;;
    "list")   redis-cli lrange "$key" 0 -1 ;;
    "set")    redis-cli smembers "$key" ;;
    "zset")   redis-cli zrange "$key" 0 -1 withscores | awk "$two_lines_to_kv" ;;
    "hash")   redis-cli hgetall "$key" | awk "$two_lines_to_kv" ;;
    "stream") redis-cli xrange "$key" - + | awk "$xrange_to_csv" ;;
    *)        echo "Unsupported: $type" ;;
  esac
done
EOF

docker exec -i immich_redis bash << 'EOF'
KEYS="immich_bull:storageTemplateMigration:*"
two_lines_to_kv='{ sub(/^[0-9]+\) /, "") ; if (NR % 2) { key = $0 } else { print key ": " $0 } }'
xrange_to_csv='/^[0-9]+-[0-9]+$/ { if (NR > 1) print output ; output = "id=" $0 ; next } { if (!key) { key = $0 } else { output = output ", " key "=" $0 ; key = "" } } END { print output }'
redis-cli --scan --pattern "$KEYS" | sort | while read -r key; do
  type=$(redis-cli type "$key")
  echo -e "\n==== $key ($type) ===="  
  case $type in
    "string") redis-cli get "$key" ;;
    "list")   redis-cli lrange "$key" 0 -1 ;;
    "set")    redis-cli smembers "$key" ;;
    "zset")   redis-cli zrange "$key" 0 -1 withscores | awk "$two_lines_to_kv" ;;
    "hash")   redis-cli hgetall "$key" | awk "$two_lines_to_kv" ;;
    "stream") redis-cli xrange "$key" - + | awk "$xrange_to_csv" ;;
    *)        echo "Unsupported: $type" ;;
  esac
done
EOF

(Depending on your NAS setup, you may need to prepend sudo to the command, like sudo docker exec .... If original command fails.)

EthanDOP•4w ago

I ran the command with sudo in the front, but it did not seem to output anything It did crash the server and send me a bunch of emails about how the container stopped through Container Manager Whatever that code did actually fully broke my Immich server....it's just constantly starting and stopping now.

Sergey Katsubo•4w ago

Sorry to hear that. The commands probably caused excessive load on Redis, especially if there are a lot of leftovers from failed/waiting jobs. I guess you already cancelled the command? Then you can try restarting Redis container.

it's just constantly starting and stopping now.

What's in the logs?

EthanDOP•4w ago

I tried restarting the NAS as a whole, but it still kept starting and stopping. I manually stopped them using Container Manager. I just tried restarting them, but it says the database is locked. I'm going to restart the NAS again and then report back!

EthanDOP•4w ago

I'm getting a lot of this in the logs:

EthanDOP•4w ago

I'm wondering if this is related to https://discord.com/channels/979116623879368755/994044917355663450/1417244276248678513 this message chain that I was going through earlier while trying to upgrade the server

Sergey Katsubo•4w ago

Check the list of all containers on NAS Maybe there are ghosts/leftovers of previous Redis docker ps -a from command line (I'm not familiar with Container Manager, perhaps it can list all containers, including the stopped ones) The ENOTFOUND redis - Immich cannot find Redis. Perhaps Redis containers is not started / is down.

EthanDOP•4w ago

This is what I see on my end for both the command and container manager

EthanDOP•4w ago

I manually stopped the containers in container manager again though so my email stopped getting flooded. The greyed out circles are manually stopped containers And docker ps -a shows that "exited about a minute ago"

Sergey Katsubo•4w ago

Interesting, immich_redis has dash instead of "immich-app" project on the screenshot

EthanDOP•4w ago

I'm unsure if it was that way before I ran the rm command from earlier What do you think the next step I should take is?

Sergey Katsubo•4w ago

- Remove immich_redis container - Start the immich-app project again and see if Redis appears in the project. In regular docker lingo it's docker compose up -d, not sure about Containr Manager

EthanDOP•4w ago

I'll stick to using ssh, I'm just using container manager as my GUI since all of this is extremely new to me

Sergey Katsubo•4w ago

docker rm -vf immich_redis

EthanDOP•4w ago

The redis container is successfully removed. Running docker compose up -d gives me a new error: "The container name "/immich_machine_learning" is already in use by container "FFFFFF". You have to remove that container to be able to reuse that name." Which was a similar error to what I got when trying to upgrade the server

Sergey Katsubo•4w ago

This may happen if they were deployed by other means, not through "docker compose". So they have "immich-app" project label defined/attached. So two options - reuse the same approach as during initial deployment - switch to pure docker compose. Means removing all other remaining containers docker rm -f immich_machine_learning immich_server immich_postgres and bringing up the stack again with docker compose up -d Do you have photo data in ./library and database files in ./postgres in your current directory where compose files are located?

EthanDOP•4w ago

Yes, I do. In ./library, there are photos in multiple places for some reason. - library/encoded-video has some videos in there, buried under multiple folders that are randomized characters from the looks of it. - There are photos in library/library/admin, which are sorted by year and month, along with xmp files for most of the photos. - There are photos in library/library/randomcharacters, which I believe are my wife's photos uploading from her phone. - Finally, there are photos in library/upload, buried under multiple folders that are randomized characters as well. In ./postgres, there are a lot of files, but I'm not sure what all they are...

Sergey Katsubo•4w ago

As expected. Great.

Sergey Katsubo•4w ago

(I've recently posted https://www.reddit.com/r/immich/comments/1ne3p7i/comment/ne4i8rm/ about Immich folders layout)

skatsubo's comment on "File location"

Explore this conversation and more from the immich community

Sergey Katsubo•4w ago

Then, if using docker compose from now on - docker rm -f immich_machine_learning immich_server immich_postgres - docker compose up -d

EthanDOP•4w ago

Okay, I am running it now! Thank you for the link to your reddit post. In the interest of learning more about this stuff, what do you mean by "using docker compose from now on?" I wasn't aware that I was using an alternative, unless you mean the container manager GUI

Sergey Katsubo•4w ago

Perhaps both approaches are about docker compose - Container Manager runs as a wrapper for docker compose, maybe with extra scripting / env vars under the hood (like https://docs.docker.com/compose/how-tos/environment-variables/envvars/#compose_project_name - if speaking about that project name column in GUI) - Therefore "pure" docker compose did not recognize the existing containers. This can be fixed - if we know what CM does, then we can set same env vars, etc. THus aligning both approaches, bringing an ability to manage the stack from CLI and from GUI

Docker Documentation

Pre-defined environment variables

Compose pre-defined environment variables

Gaming

Programming

Enabling Storage Templates ended in jobs not completing

Did you find this page helpful?