One-Time Short Term 100 TB data migration job
I need to do a roughly 100 terabyte data migration from a third-party CDN to an R2 bucket in a relatively short period of time. The images will need to be resized and compressed before uploading to R2. Is it possible to use Cloudflare Workers to maximize the short-term bandwidth and compute power of this migration, and if so how would I go about doing that?
2 Replies
The idea to use Workers is inspired by the functionality of the S3 to R2 super slurper, but in this case, all we have for accessing the CDN we are pulling images from are image URLs which will expire some time next month.
I'm pretty sure you could build such a pipeline with Queues.
Queues scale up to a maximum of 20 concurrent consumers so if you're writing an efficient pipeline it could work. But you have to make the math yourself if its fast enough.
Maybe you can request a limit increase for the concurrency but this might require an ENT plan. But if were talking about 100TB of storage you probably have one anyways.