My disk failed - help to recover
My drive with Ente running on it has just failed. 🙁
I keep backups of my compose and museum YAML files plus I have daily backups of the PGSQL DB. I also have my MinIO replicated to two other disks.
I recently asked another question about the replication, as the MinIO buckets are not identical sizes.
ie my wasabi-eu-central-2-v3 is 631.3 GiB and scw-eu-fr-v3 is 624.8 GiB and my b2-eu-cen was about 20GB larger than those two. So, to recover, can I just promote one of the other storage buckets? What was in the extra space it was using? Assuming I can do that, will it just start replication again to a new disk? Whats the process?
ie my wasabi-eu-central-2-v3 is 631.3 GiB and scw-eu-fr-v3 is 624.8 GiB and my b2-eu-cen was about 20GB larger than those two. So, to recover, can I just promote one of the other storage buckets? What was in the extra space it was using? Assuming I can do that, will it just start replication again to a new disk? Whats the process?
5 Replies
After a disk failure, you can promote one of the other storage buckets. To enable replication after configuring all 3 storage buckets, set
replication.enabled to true in museum.yaml:
Replication is disabled by default, with only the first bucket (b2-eu-cen) being used. If you'd like to set one of the other pre-defined buckets as the primary bucket, use the s3.hot_storage.primary option. See the discussion on reliability for more information.
Replication works only if all 3 storage buckets are configured (2 hot and 1 cold storage). For more information, check this discussion.
-# If your issue is resolved, feel free to use the /solved command to close this thread. If you'd like to ask me another question use /docsearchente
Reliability and Replication
Ente keeps 3 encrypted copies of your data across EU data centers—including an underground facility—to ensure durability and protect against data loss.
GitHub
Self hosting: replication not working/unreliable when only two buck...
Hello there! I've been playing around with self hosting Ente and love it so far. I've tried to set up replication between the primary b2-eu-cen bucket and scw-eu-fr-v3, using Backblaze and ...
🔍 Similar Solved Posts
Found some similar posts that might help:
📋 Check these out:
Searched 58 solved posts
My replication was already set to true, hence why i have the two other copies.
So, do i change it to false, then point the b2-eu-cen to the address of wasabi-eu-central-2-v3 to get it running again? Then change the flag back to true once there is a third MinIO again?
Quack :lilducky:
Well, I have just gone and done these things:
- renamed bucket from wassabi to "b2-eu-cen" on my seconday MinIO server.
- built a new Ente instnace in docker
- Restored the DB backup from 26/10
- changed museum.yaml so that replication = false.
- Pointed Proxy to new IP
It all seems to be working from the UI, however in the logs there are some things that are concerning.
Lots of these:
museum-1 | WARN[14437]handler.go:49 Error Request failed error= --- at /etc/ente/pkg/controller/filedata/controller.go:102 (Controller.InsertOrUpdateMetadata) ---
museum-1 | Caused by: INVALID_VERSION : version mismatch expected 1761265588169958, found 0 req_id=ser_EFAvlQVD4idzLr3tlIFPaH user_id=1580559962386438
and big solid chunks of errors similar to:
---
museum-1 | Caused by: object not found file_id=10102905 latest_bucket=b2-eu-cen objectKey=1580559962386438/file-data/10102905/mldata req_id=ser_HqkMs3trKxpfU2ZwtXY8n4
museum-1 | ERRO[14690]controller.go:248 func1 error fetching object: 1580559962386438/file-data/10102656/mldata --- at /etc/ente/pkg/controller/filedata/controller.go:308 (Controller.fetchS3FileMetadata)
Im guessing the latter block is to do with the machine learning data and it must be data that isnt replicated between primary and secondary S3 buckets. Will that just be regenerated?
The first error, I dont know what it means, I dont know if it will just sort itself out, or of there is more work I should be doing on it?
Anyone know what these errors mean and what I should be doing?