Best practices for large-scale R2 backup (≈250 TB, long-running)
🌍CDN✅Solved
Hey all ! (I'm using a translator)
I’m planning a one-time large backup operation from Cloudflare R2 to another object storage (Backblaze B2), and I’d really appreciate some guidance to make sure I handle this as safely as possible.
Context:
- Around 250 TB total
- Multimedia content uploaded by my customers, mostly delivered using HLS (m3u8 playlists with many small .ts segments)
- The R2 bucket is strictly read-only during the operation (no deletes, no sync)
- Using rclone in copy mode only
Important detail: this operation is specifically meant to create a backup. At the moment, I don’t have any other copy of this data, so my main concern is avoiding anything that could trigger throttling, temporary blocks, or account issues during the transfer.
Planned approach:
- Stable and continuous bandwidth (roughly 60–120 MB/s total, no bursts)
- Limited concurrency (2–3 parallel rclone processes, low transfers/checkers)
- Folder-by-folder copy (per logical content unit), no global bucket scans
- No fast-list, no aggressive retries
Goal:
- Keep traffic predictable and non-abusive
- Safely complete a full backup without putting the source data at risk
- Minimize the chance of interruptions during a long-running transfer (~2–3 weeks)
Before starting, I just want to double-check:
- Is this approach aligned with R2 best practices for large-scale backup operations?
- Are there specific limits or patterns I should be extra careful about when doing a long-running backup like this?
Thanks a lot
I’m planning a one-time large backup operation from Cloudflare R2 to another object storage (Backblaze B2), and I’d really appreciate some guidance to make sure I handle this as safely as possible.
Context:
- Around 250 TB total
- Multimedia content uploaded by my customers, mostly delivered using HLS (m3u8 playlists with many small .ts segments)
- The R2 bucket is strictly read-only during the operation (no deletes, no sync)
- Using rclone in copy mode only
Important detail: this operation is specifically meant to create a backup. At the moment, I don’t have any other copy of this data, so my main concern is avoiding anything that could trigger throttling, temporary blocks, or account issues during the transfer.
Planned approach:
- Stable and continuous bandwidth (roughly 60–120 MB/s total, no bursts)
- Limited concurrency (2–3 parallel rclone processes, low transfers/checkers)
- Folder-by-folder copy (per logical content unit), no global bucket scans
- No fast-list, no aggressive retries
Goal:
- Keep traffic predictable and non-abusive
- Safely complete a full backup without putting the source data at risk
- Minimize the chance of interruptions during a long-running transfer (~2–3 weeks)
Before starting, I just want to double-check:
- Is this approach aligned with R2 best practices for large-scale backup operations?
- Are there specific limits or patterns I should be extra careful about when doing a long-running backup like this?
Thanks a lot
Welcome to the official Cloudflare Developers server. Here you can ask for help and stay updated with the latest news
86,942Members
Resources
Was this page helpful?
