Rclone sync library folder: "Duplicate object found in source - ignoring"
I'm trying to sync my library folder to OneDrive to have a file level copy of the photos (& postgres, but less important to me). The sync is slow, but it's on the OneDrive account now. What I don't understand is why rclone gives me thousands of this notice:
06-25/20080625P1010311_1.jpg.xmp: Duplicate object found in source - ignoring
Immich is supposed to have deduped everything? If i login to my debian server, I only see 1 item in my library folder with that name, but I do see 06-25/20080625P1010311_1+1.jpg.xmp
Is the +1 something Immich does? Something which rclone does not like?
14 Replies
:wave: Hey @wokkieman,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
2. :blue_square: read applicable release notes.
3. :blue_square: reviewed the FAQs for known issues.
4. :blue_square: reviewed Github for known issues.
5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.I’m confused what direction is the sync going in?
My guess is you have duplicate named files on one drive as they may not support upper and lower case exts as separate files. Since it’s Microsoft
The storage template will make all things lowercase in the next version
Either that or one drive is getting confused about the +1
The +1 is when you have two files of the same name. You likely have uploaded two versions of the same file with different metadata (which aren’t exact duplicates )
it should go from my local server (running latest Immich) to OneDrive. But I will try some things with or without the +.
Or it's just time for me to find a normal back up plan instead of onedrive 🙂
yes, you really should be using a real backup tool - for example restic or borg
you can stil use onedrive as an endpoint for restic
my idea was keeping this purely file level. So I don't need any tools and could just browse the photos if I need something quickly. Maybe I need to throw that idea overboard.
Locally I have PBS
bad plan IMO, no versioning, no encryption, etc
to add to that list, also very slow with 700GB and a ton of small files 🙂
but thanks for the quick feedback. Let's adjust in the weekend 🙂
Does this mean that all filenames will also be converted to lowercase? I've been looking around to see if there is a way to do some processing on uploaded filenames to convert them to lowercase and ensure that any special characters to ensure that filenames will play nice with all file systems.
No. Only the extension is lower case
However you can easily setup the storage template to not use the original filename. You can use just UUID and timestamp for example
That's a good point... I totally missed that 😅
I'll probably just use
{{assetIdShort}} for now. Maybe I will look into putting something custom into my reverse proxy that would modify the uploaded filenames at some point
Although, I don't like that it wouldn't work for anything I have already uploaded... I guess I would have to manually run a script on the Immich docker container for something like that, but then would that mess up any DB metadata linking? 🤔never modify the Immich library filesystem directly
Good to know, I thought that might mess some things up. Glad to have the confirmation
You can just run the storage migration job? I’m not sure what you mean by already uploaded
The storage migration job works great for using the UUID like you mentioned! I've actually already done that.
I was just considering other alternatives that would be able to maintain more human readable file names, but enforce some rules that ensure the filenames play nice with different file systems.
I have auto-upload setup on my phone to automatically upload photos from there. But I also will sometimes upload photos from my computer or photos that someone else shared with me. So, I don't want to make any assumptions about the filenames of the photos being uploaded. It's definitely not a big deal, but I was thinking it would be nice to use the uploaded filenames, but just do some processing on them to replace spaces/special characters with
- or _ and convert the names to lowercase. I don't think additional custom processing like this is supported by the migrator unless I am missing something.