What is the robust way to import 2 TB of photos?
I am running Immich in a docker container on TrueNAS. A family member gave me ~2TB of files, most of which is movies/pictures. These files have been copied to my NAS, but live in a chaotic folder structure with many duplicates.
I have tried uploading the files with both immich-cli and immich-go. Both crash. Immich-cli copies around 500 GB of files before it exhausts the memory in the docker container (16GB). immich-go loses connection to the API after ~300 TB.
The problem is mainly that there is no easy way to resume. I have not tried importing as an external library because I would prefer to run a storage template job after import.
Is the only reasonable solution to just look at immich-go's log and manually exclude the imported folders? Unfortunately it seems like just restarting it takes too long.
Happy to hear any tips!
5 Replies
:wave: Hey @frisby,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJAChecklist
I have...
1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
2. :blue_square: read applicable release notes.
3. :blue_square: reviewed the FAQs for known issues.
4. :blue_square: reviewed Github for known issues.
5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like
fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.GitHub
immich-app immich · Discussions
Explore the GitHub Discussions forum for immich-app immich. Discuss code, ask questions & collaborate with the developer community.
FAQ | Immich
User
GitHub
Issues · immich-app/immich
High performance self-hosted photo and video management solution. - Issues · immich-app/immich
Hi, would recommend batching the upload. For example run it once on each subfolder
The server will reject exact dupes
Also, try pausing the background jobs while you import. I think odds are the server is just getting overloaded by those
This is a good idea, that's probably part of why immich struggles, I will try this. Thanks!