Bulk (re)write of xmp sidecar files
If I understand correctly, both sidecar sync and discover extract xmp data to the database. Changing GPS date tags in immich writes to the sidecar, on save/commit. If the sidecar is lost, is there anyway to bulk sync from database to sidecars ?
10 Replies
:wave: Hey @john88880152,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
2. :blue_square: read applicable release notes.
3. :blue_square: reviewed the FAQs for known issues.
4. :blue_square: reviewed Github for known issues.
5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.Interesting question. At first glance, after skimming through the code, there is no ready-to-use option in UI. Maybe it can be accomplished with extra scripting and API or running custom code inside the container.
Any particular use case or issue you are trying to solve?
I think the use case is poor planning 😊. I am currently moving between shotwell moments in nextcloud and immich ( as I just tried it, it's Great! ). Messing around with test migrations. All external. If I do some tagging / gps / date changes, in immich, they are in the database. If I mess up or redirect the docker mappings to a clean version of the photo filesystem. ( Where nominally most of the photos are same name same place) Then all the xmp files are lost. I would like to be able to regenerate, or "sync" from database to xmp, otherwise I can't migrate in the future.
I should have said in that in normal use, the sidecars are created, it's only when they are lost, they are never recreated.
Immich isn't expecting to deal with things disappearing from the filesystem, the normal 'solution' when files go missing would be to restore them from a backup
Totally understand/agree. However, things happen. The clearer usecase I should of stated:
I have a master Library of photos. I make a working copy of the Library as an external for Immich. I do lots of stuff. The working directory is lost or partially corrupted. I swear a bit. I can recreate the working directory from the master easily, either completely or partially. BUT some or all of the xmp files are lost, even if it is only the ones between any backup strategy. I now have the Photos, and the tags/gps in the database, but no way (?) to recreate or even detect the missing xmp sidecar files. I am open to any pointers to extract database to xmp manually if they allready exist. Thanks for the replies.
I imported my photos from synology photos. The great majority of my photos do not have sidecars. The database is updated faithfully but I can't say the same for sidecars except for photos not imported. The reason why I switched from Synology Photos to Immich was because of the sidecars. Synology photos is proprietary and does not use sidecars. Digikam and other photo apps can read sidecars, I switched to Immich because of the sidecars and since it was open source. I don't want my pictures and videos hostage to an app that will be dropped for lack of interest or is proprietary. Using Sidecars is peristent storage. Immich, walk before you run, please fix this very serious problem. @john88880152 I share your frustration.
The metadata is all within the original image itself. I’m not sure what makes this a “very serious problem”
The sidecars are created by immich if metadata is edited in any way
^^ This.
The doc explains handling of sidecars pretty well - https://docs.immich.app/features/xmp-sidecars
If you later update tags, rating, or description in the web UI, Immich will update both the database and the copied .xmp file to stay in sync.(External libraries mounted read-only have their limitations, this is also in the doc)
The original metadata is in the original image and updates the sidecars most of the time. I have images I imported where sidecars are not created when the description as an example is updated. Subsequent changes within Immich will not update the original file itself and that's fine. This is a similar policy Google uses. Synology Photos will update the original images but not videos. I have many images and videos without sidecars though the database contains metadata. There are a number of reasons, I won't get into, why it would be good to have the means of ensuring the sidecars contain the same data as the database - namely portability. A Synchronization button would be awesome. I love Immich, it's just a growing pain. I see so much development and the top dogs of Immich are doing a fantastic job.
Following this thread I wrote immich-metadata-exporter (github) to dump asset metadata from the Immich database and write to XMP sidecars (update existing or create missing) using exiftool under the hood, similarly as Immich does.
It's basically a proof of concept script written in bash + sql. See the project README for details.
Try it with
bash export.sh --preview - dry run, writes sidecars to the current working dir instead of original locations.
If you manage to run it (haha) then let me know, your feedback is appreciated.