Migrating from external library to default
Hey Immich Community,
I have a large library of photos (320,000) and videos (17,500), this is an External Library in Immich as I am currently using PhotoPrism for part of my workflow, I am looking to remove PhotoPrism from my workflow and have the Immich External library become a default library.
My current immich-server docker-compose:
immich-server:
container_name: immich_server
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
restart: always
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
- /etc/localtime:/etc/localtime:ro
- /mnt/chamber/photoprism/originals:/mnt/media/chamber-photos:ro
env_file:
- stack.env
depends_on:
- redis
- database
networks:
- traefik
- immich_internal
ports:
- 2283:2283
labels:
- traefik.enable=true
- traefik.http.routers.immich.rule=Host(
immich.hidden.hidden)
- traefik.http.services.immich.loadbalancer.server.port=2283
- traefik.http.services.immich.loadbalancer.server.scheme=http
- traefik.http.routers.immich.entrypoints=websecure
- traefik.http.routers.immich.tls.certresolver=letsEncrypt
The photo library is located at /mnt/chamber/photoprism/originals
My current workflow involves my wife and I using the PhotoSync app to transfer media files to my NAS, I want to keep part of the process for now as we like the customisation vs Immich app for backing up. The PhotoSync app moves media assetts to /mnt/chamber/photoprism/import, from there PhotoPrism imports them and sorts them into /mnt/chamber/photoprism/originals/[year]/[month]/[file_name]. I believe I can achieve the same behaviour using Immich storage template.
I am looking to understand how I migrate this library from 'external' to 'default' without loseing faces, tags, albums etc within Immich. Based on my research I likely need to use the CLI, which I am OK with but looking to build a bit more confidence on the method.
Thanks24 Replies
:wave: Hey @Jack Brown,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:I am looking to understand how I migrate this library from 'external' to 'default' without loseing faces, tags, albums etc within Immich.You can't There is no process to move from external to internal without duplicating the asset If you mean to save tagged people I think the only way would be to migrate about half, let ML run it's course so it tags the people with their old assets in place, so you at least save the people There's no saving tags on the picture itself, unless you have that in metadata?
I think the key things would be albums, named faces and tagged faces
I mean, in xmp files
Did you do a lot of manual tagging?
not a huge amount
So to clarify what you can and can't keep
people are made of recognized faces from assets, if you remove all assets at once then the person disappears
If you migrate it slowly, or the recognized faces are stored outside of immich, for instance in exif/xmp data, then those are saved
Albums will simply not contain any assets anymore if you move libraries, so those will need to be manually recreated, or if you stored them on disk somehow you can import a folder as album
OK thank you, I guess for albums I could write a script and use the API to recreate the albums using something like original file name field?
That would work
If you feel comfortable scripting you could even make a whole migration script that replaces one asset at a time :p
Do you have a view on the high-level steps involved? I am comfortable in scripting but would need to get more familair of what would need to be done and in which order within Immich
I think real basic would be
- Get overview of assets in the external library
- Iterate over them:
-- Get albums asset is in
-- Upload copy to internal library
-- Put asset in albums
-- Repeat
The facial recognition should work automatically and group them with the existing people
I see, so at the end of the script there would be duplicate of every assett and the external library can then be removed?
Or as part of 'upload copy step' it gets removed from external location?
It shouldn't get removed unless you do a move instead of a copy 😛 and yes my intent was to decouple the external library at the finish
I think a faster way might be to just yeet all assets to upload, and then use the API to get the assets in the albums, get their hashes, look for matching assets in the internal library and add those 🤔
Yeah I think you may be right,,,,,so import everything so everything is duplicate. Once imported use API to look at old album items and then add same new assets using hash/original file name then get rid of external
The discussed approach (using API) is definitely the proper/recommended way of doing such migration.
Yet there is another, hacky and not recommended/supported way by modifying records in the database. It was discussed in https://github.com/immich-app/immich/discussions/19853
I was wondering if there is a way I can migrate them from an external library to look like they were uploaded, just modifying some data in the database.and drafted in https://github.com/skatsubo/immich-library-external-to-internal Only try it on a throaway instance because it is just a proof of concept.
I have used CLI to import assets and then I have created a amateur python script to take care of re-adding assets to albums. Do I then just 'delete' the external library in Immich settings? Will that remove all database entries etc for those external assets?
I'm not sure if it's instant delete or they get put in the trash first
But either way, yes that's the way to go
and then empty the trash
UI confirmation dialog
Are you sure you want to delete this library? This will delete 1 contained asset from Immich and cannot be undone. Files will remain on disk.Immich log Postgres queries
Thank you
I am seeing a lot of these in the PG logs, will try debug / research
ERROR: duplicate key value violates unique constraint "UQ_assets_owner_checksum"
This is the server-side duplicates prevention.
Are you uploading files that are already in the internal/default/user library, maybe upon the recent migration ext->default?
Im currently still performing the external library -> interna;/default/user/library import, will duplicates be detected between externa;/internal library and is it safe to ignore these for now as I will eventually remove the external library?
I suggest to have a look at what assets it is complaining about
Smth like
The "where" part is copy-pasted from
Thanks I used your command to find the asset original file name, and I can find the asset in DB twice by using original file name
select *
from asset
where "originalFileName" = '20141004_114648_EB874FEE.jpg';
one with filepath of my external library and one with the new /data/upload
External lib has
Key ("ownerId", "libraryId", checksum)
Internal lib has Key ("ownerId", checksum) as in the error message.
I would guess the script already created asset record with /data/upload (internal), then the script tried to create it again and errored out.