cmaddox
CDCloudflare Developers
•Created by Shankar_sq on 4/16/2025 in #r2
I recently used slurper tool to move
Ok got some more context on this one.
1. This being an infrequent access bucket means that any data uploaded (failed upload or not) has a minimum retention of 30 days, which is currently ~7TiB. After 30 days, the failed uploads will be dropped.
2. While the end result was 5-7% objects failing, we do retry each part if it fails. This means you could end up uploading each part multiple times if it fails halfway through a couple times. We rety each part a maximum of 3 times.
3. The 11TiB I believe is incorrect on the dashboard is a bit perplexing. If you navigate into the bucket, does it still show 11TiB?
6 replies
CDCloudflare Developers
•Created by Shankar_sq on 4/16/2025 in #r2
I recently used slurper tool to move
Dug a little more into this.
TLDR; the data stored for the account is the 2.7TB and ~17000 object as mentioned. The extra data that is showing is from the failed uploads, which should be automatically cleaned up when the multipart abort object lifecycle rule runs for them (likely today or tomorrow based on when you ran the migrations).
I'll keep an eye on your account to ensure that occurs.
6 replies
CDCloudflare Developers
•Created by Shankar_sq on 4/16/2025 in #r2
I recently used slurper tool to move
Hey there, the 7TB looks to be the correct amount of data for that account.
A few questions:
1. How many objects roughly are expected for the bucket?
2. Do you see any objects in the bucket that have sizes that are much larger than expected?
3. Is your script pulling every object in your bucket or a particular prefix?
I did a few spot checks and object sizes appear to be correct based on what was received from the original source bucket.
6 replies
CDCloudflare Developers
•Created by oof2win2 on 4/10/2025 in #r2
hmm it still isn't doing anything. found
Ah, that explains it. Supabase is not fully supported for slurper at the moment, since they do not accept encoded object keys.
5 replies
CDCloudflare Developers
•Created by oof2win2 on 4/10/2025 in #r2
hmm it still isn't doing anything. found
Hmm, interesting. We will take a look and try to reproduce this error on our side. Can you confirm what provider you are migrating from?
5 replies
CDCloudflare Developers
•Created by oof2win2 on 4/10/2025 in #r2
hmm it still isn't doing anything. found
Took a peek at the logs for your migration and we are getting 403s when sending HEAD requests for a lot of the objects in the source bucket. Are you able to manually test the tokens you used?
5 replies
CDCloudflare Developers
•Created by Joooorriτ on 4/2/2025 in #r2
Hi! I tried to migrate a dataset but it
No problem!
7 replies
CDCloudflare Developers
•Created by Joooorriτ on 4/2/2025 in #r2
Hi! I tried to migrate a dataset but it
You can rerun it with overwrite turned off and it will skip any files that already exist in the bucket.
7 replies
CDCloudflare Developers
•Created by Joooorriτ on 4/2/2025 in #r2
Hi! I tried to migrate a dataset but it
Hey there 👋 it seems we are missing some validation....
Rerun your migration but leave off the trailing
/ on the endpoint you enter
So just https://<account-id>.r2.cloudflarestorage.com/<bucket>7 replies
CDCloudflare Developers
•Created by Valeriu Paloș on 3/7/2025 in #r2
Hi!
Definitely could be something with the special characters, but I would expect something other than a 403 if that were the case. We'll look into it regardless and thank you for the feedback!
20 replies
CDCloudflare Developers
•Created by Valeriu Paloș on 3/7/2025 in #r2
Hi!
It looks like most of the failures received a 403 status code from the source bucket when we attempted to get the object. For some extra data, did you use the same credentials for the rclone migration as the super slurper migration?
20 replies
CDCloudflare Developers
•Created by Valeriu Paloș on 3/7/2025 in #r2
Hi!
That's fine with me 👍
20 replies
CDCloudflare Developers
•Created by Valeriu Paloș on 3/7/2025 in #r2
Hi!
Woohoo! Let me know if you run into any other issues.
20 replies
CDCloudflare Developers
•Created by Valeriu Paloș on 3/7/2025 in #r2
Hi!
@Valeriu Paloș the fix is now live! Would you mind trying your migration again?
Fair warning, Scaleway is not a provider we have explicitly tested with, but I took a peek at their documentation and it looks like it should work.
20 replies
CDCloudflare Developers
•Created by Valeriu Paloș on 3/7/2025 in #r2
Hi!
Hey there, we have a fix going out soon for some of the S3 compatible APIs. I can update you here when it is live and you can give a retry.
20 replies
CDCloudflare Developers
•Created by jappit on 2/26/2025 in #r2
we also tried using an access key/secret
You need both get and list permissions.
https://developers.cloudflare.com/r2/data-migration/super-slurper/#amazon-s3
11 replies