Hi there! I'm running a Docker container on a VPS. I want to write a shell script that can put fil

Hi there!

I'm running a Docker container on a VPS. I want to write a shell script that can
put files from the container into a Cloudflare R2 bucket. I've read through the
docs, and it seems like my best options are: wrangler, rclone, aws CLI, Workers API.

Is it not possible to simply use a curl command to a Cloudflare REST API
endpoint? For example:

curl -X PUT "https://${ACCOUNT_ID}.r2.cloudflarestorage.com/${BUCKET_NAME}/${object_key}" \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/octet-stream" \
  --data-binary "@$FILE_PATH"

curl -X PUT "https://${ACCOUNT_ID}.r2.cloudflarestorage.com/${BUCKET_NAME}/${object_key}" \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/octet-stream" \
  --data-binary "@$FILE_PATH"

I'd prefer to do something like this, rather than add

wrangler

wrangler

rclone

rclone

aws

aws

to my Docker container, or spin up a Cloudflare Worker. Thanks in advance
for the help!

Bbennycondon Hi there! I'm running a Docker container on a VPS. I want to write a shell scr...

Chaika•2/28/25, 2:12 AM

The r2.cloudflarestorage.com endpoints are just for S3.
There is an undocumented REST API that Wrangler uses, but it has limits like ~300 mb upload size, and it's slow as it has to go through Core/not processed at edge, plus it may change at any time and break since it's undocumented
Using one of the many s3 libs or rclone which use the s3 compat api would work well. You could write your own simple worker to use the Workers API, but would have to deal with multipart if files are bigger then 5 GB, which would be easier with a lib or rclone

CChaika The r2.cloudflarestorage.com endpoints are just for S3. There is an undocumented...

bennycondonOP•2/28/25, 2:15 AM

Thanks Chaika for all the help! Is there anything you can point me to on the undocumented REST API that Wrangler uses? The limits on size and speed don't bother me. If not, I'll just use rclone

Chaika•2/28/25, 2:15 AM

opps the limit on wrangler/rest is ~300 mb or so, not 500 mb

Bbennycondon Thanks Chaika for all the help! Is there anything you can point me to on the und...

Chaika•2/28/25, 2:17 AM

there is no documentation for the undocumented api ;p
I explained it more to show why it's a bad idea to use wrangler as well, I would just use rclone.

CChaika there is no documentation for the undocumented api ;p I explained it more to sh...

bennycondonOP•2/28/25, 2:18 AM

Haha that makes sense! I'll just use rclone. Thanks again for all your help, I really appreciate it.

Lone Ranger•2/28/25, 1:02 PM

Is their no way to skip time for big file?

Lone Ranger•2/28/25, 1:02 PM

very bad

Lone Ranger•2/28/25, 1:02 PM

Where do I complain about this?

Original message was deleted

Lone Ranger•2/28/25, 1:13 PM

really? you mean page rule?

velioglu•2/28/25, 1:29 PM

Hey, while saving chunks for multipart uploads, I'm sporadically getting 502 and 503 errors. I suspect this might be related to request throttling, as the error rate spiked significantly after increasing my traffic from 2K to 10K/day upload. When I retry uploading a chunk, it usually succeeds. I've checked the documentation but couldn't find any relevant details. I came across some forum threads, like https://community.cloudflare.com/t/r2-intermittent-500-errors-on-put/485134, but none seem to offer a clear resolution. Does anyone know why these intermittent 502/503 errors are occurring and how I can resolve them?

Original message was deleted

Lone Ranger•2/28/25, 1:33 PM

how? can you help me?

velioglu•2/28/25, 1:39 PM

I'm uploading to multiple buckets and don’t expect to upload 300 chunks to the same one. Does the 300 concurrent request limit apply across all my buckets collectively, or is it per bucket? In other words, if I send 300 concurrent requests spread across different buckets, would I still hit that limit?

Original message was deleted

Lone Ranger•2/28/25, 1:40 PM

ok thanks

velioglu•2/28/25, 1:40 PM

is there any other limit across buckets?

Original message was deleted

Lone Ranger•2/28/25, 1:51 PM

wow Thanks. It starts working magically.

Nick•2/28/25, 6:26 PM

Does anyone know, if I begin an upload then abort it, will the bytes end up in the bucket, or automatically cleaned up?

Nick•2/28/25, 6:26 PM

I’ll test it myself later if not, just wondering

NNick Does anyone know, if I begin an upload then abort it, will the bytes end up in t...

Chaika•2/28/25, 6:29 PM

Multipart uploads are aborted after 7 days by default (can modify in settings under lifecycle rules). If it's just a normal PUT that gets killed shouldn't go anywhere

Nick•2/28/25, 6:29 PM

Sorry I’m just talking about a PUT

Nick•2/28/25, 6:29 PM

When you say shouldn’t go anywhere you mean the incomplete upload never makes it into the bucket?

NNick When you say shouldn’t go anywhere you mean the incomplete upload never makes it...

Isaac McFadyen•2/28/25, 7:15 PM

If a regular PUT is aborted halfway through the data transferred is just discarded, it never goes anywhere (including the bucket)

Nick•2/28/25, 7:17 PM

Thanks

Vi•3/1/25, 1:52 AM

This may be stupid but when serving files via r2 and the cloudflare cdn, is it really unlimited bandwidth? Like truly uncapped? If so, why doesn't youtube or netflix just host stuff on r2? Wouldn't it just massively bring down server costs?

Zora•3/1/25, 3:08 AM

youtube or netflix probably get the storage for cheaper and they have their own ways to save bandwidth

Zora•3/1/25, 3:08 AM

storage on r2 is pretty expensive compared to others

Vvarundeva Im getting error while uploading file to r2 `[Error: getaddrinfo ENOTFOUND rakth...

Triple•3/1/25, 10:37 AM

here's mine

nehatkhan•3/1/25, 11:01 AM

hello guys need help uploading images to R2 all at instead of uploading one by one. please suggest the way to upload images at once.

Nnehatkhan hello guys need help uploading images to R2 all at instead of uploading one by o...

Jadugar_Jaggu•3/1/25, 11:15 AM

you can upload a folder using cloudflare dashboard

Original message was deleted

nehatkhan•3/1/25, 11:17 AM

I thought of doing that using promise.all but what if some images failed to upload and it throw error. so the images will get uploaded will be there for life till I manually delete the uploaded image manually. what is the best approach to handle this scenario.

basically I have a travel package model where I am uploading images and package details.

JJadugar_Jaggu you can upload a folder using cloudflare dashboard

nehatkhan•3/1/25, 11:18 AM

sir have to do it with code for client.

Vvarundeva Im getting error while uploading file to r2 `[Error: getaddrinfo ENOTFOUND rakth...

Jadugar_Jaggu•3/1/25, 11:19 AM

cloudflarestorage.com subdomains of this domain isn't accessible with some indian ISPs like Jio, if you are locally using Jio or related ISP you have to use vpn or proxy

Nnehatkhan I thought of doing that using promise.all but what if some images failed to uplo...

Jadugar_Jaggu•3/1/25, 11:20 AM

You can do promise all with cloudflare workflows steps, it has auto retry mechanism

JJadugar_Jaggu You can do promise all with cloudflare workflows steps, it has auto retry mechan...

nehatkhan•3/1/25, 11:21 AM

nice sir. could you please share article or doc link to achieve this

Vvarundeva How can i do that? Any guidance? Because if same happens that should be done in...

Jadugar_Jaggu•3/1/25, 11:23 AM

You don't have to use VPN in VPS server cause they will be able to access it, as they do not have any content restrictions

Jadugar_Jaggu•3/1/25, 11:24 AM

it's only ISP Jio who occasionally bans cloudflare domains and ips

Nnehatkhan nice sir. could you please share article or doc link to achieve this

Jadugar_Jaggu•3/1/25, 11:25 AM

https://developers.cloudflare.com/workflows/

It's currently in beta

Cloudflare Docs

Cloudflare Workflows · Cloudflare Workflows docs

Workflows is a durable execution engine built on Cloudflare Workers. Workflows allow you to build multi-step applications that can automatically retry, persist state and run for minutes, hours, days, or weeks. Workflows introduces a programming model that makes it easier to build reliable, long-running tasks, observe as they progress, and progra...

JJadugar_Jaggu https://developers.cloudflare.com/workflows/ It's currently in beta

nehatkhan•3/1/25, 11:26 AM

sir is this like queue system?

Nnehatkhan sir is this like queue system?

Jadugar_Jaggu•3/1/25, 11:26 AM

No you can read more about it on the official site

JJadugar_Jaggu No you can read more about it on the official site

nehatkhan•3/1/25, 11:26 AM

okay sir thank you so much for the help and time.

Nnehatkhan I thought of doing that using promise.all but what if some images failed to uplo...

gruntlord6•3/1/25, 3:37 PM

Alternative method would be to clear that directory or bucket after a specific amount of time, depending on the use case.

I have a system for user uploads that gets moved after they are processed and otherwise they are deleted after 7 days by the bucket retention.

In other projects I take a different approach

VVi This may be stupid but when serving files via r2 and the cloudflare cdn, is it r...

gruntlord6•3/1/25, 3:39 PM

Not for video streaming, they have a separate service for that.

But for other files yes it's purely the amount of reads/writes you use that gets billed

If you use the cdn caching that basically becomes negligible so you really just pay for storage at that point

gruntlord6•3/1/25, 3:40 PM

Yea but I don't think video streaming is allowed on the cdn