What issues did you have with b2 for hot storage

Z

zegevlier•3/21/24, 11:00 AM

There's value in not having to pay thousands of dollars up-front vs having to pay a higher montly fee.

Z

zegevlier•3/21/24, 11:01 AM

Plus, if you're dealing with your own setup you need to fix it when it breaks, versus a managed service that someone else has to fix. You can see that either as an upside or a downside

Z

zegevlier•3/21/24, 11:02 AM

For things like backups there's also hetzner storage boxes, which are much much cheaper than most S3 providers

Z

zegevlier•3/21/24, 11:03 AM

Yup, exactly. Original copy on device, second copy on local nas, off site on hetzner storage box.

RRw quick question here anybody here ever consider wasabi or blackblack b2 or tecent...

Z

Zora•3/21/24, 12:57 PM

Wasabi: $6.99/TB/month, has a 90 days minimum storage duration and 1 TB minimum storage charge, no meaningful bandwidth alliance benefit, free egress up to the amount stored each month

Backblaze B2: $6/TB/month, rate limits your account after a small number of requests per second, they refuse to increase the limit

Tencent COS: $20-24/TB/month, has a 10 Gbps bandwidth (not traffic) limit and progressively throttles you to hundreds of kilobytes per second for exceeding it, it takes up to a few weeks for them to "evaluate" and increase the limit, in the meantime you'll continue to be throttled

Alibaba Cloud OSS: $16-17/TB/month, has a 5 Gbps bandwidth (not traffic) limit and progressively throttles you to hundreds of kilobytes per second for exceeding it, it takes up to a few weeks for them to "evaluate" and increase the limit, in the meantime you'll continue to be throttled

Source: Tried all of them, I don't recommend using Tencent or Alibaba since they're more expensive than R2 in terms of price per TB. My use case isn't video streaming but it does around 70 requests per second and pushes 40-65 Gbps

Original message was deleted

Z

Zora•3/21/24, 1:07 PM

Can't say much about video streaming but I've pushed tens of petabytes each month on B2 and cloudflare didn't say anything

Z

Zora•3/21/24, 1:11 PM

It was one of those "only one way to find out" things since everywhere I looked about that was only speculation and not real-world usage

Bbun What issues did you have with b2 for hot storage

Z

Zora•3/21/24, 1:14 PM

They rate limit you

B

bunOP•3/21/24, 1:20 PM

Nice

Original message was deleted

C

Chaika•3/21/24, 3:14 PM

logpush http

C

Chaika•3/21/24, 3:14 PM

If it's through an R2 Custom domain, that is

Original message was deleted

C

Chaika•3/21/24, 4:34 PM

you can have it do all, or a set sample

C

Chaika•3/21/24, 4:34 PM

eitherway though, besides the cost for Enterprise it's free, so "cheap" in a way

R

Rw•3/21/24, 6:50 PM

another question i got in mind, for r2 egrees since its free what's the point connect to a domin for cloudflare cdn cache, i mean cloud objective are already cdn based right especially within the cdn platform

K

kian•3/21/24, 6:51 PM

Saving on operations, like GetObject

R

Rw•3/21/24, 7:36 PM

so since i already have argo on my web if i am also using R2, is R2 going to use argo for argo smart routing and charge ?

RRw another question i got in mind, for r2 egrees since its free what's the point c...

Z

Zora•3/21/24, 9:29 PM

all the features that zones have, like caching and waf

RRw so since i already have argo on my web if i am also using R2, is R2 going to use...

K

Karew•3/21/24, 9:39 PM

If you have Argo enabled, it does affect R2 and charge you for R2 bandwidth too

K

Karew•3/21/24, 9:41 PM

If you have an R2 custom domain, that is already Cloudflare <-> Cloudflare communication so I wouldn't really feel like enabling Argo for R2 is worth it

Z

Zora•3/21/24, 9:46 PM

Keep in mind though that custom domains use whatever zone's plan you have (free, pro, biz), presigned urls use enterprise

Z

Zora•3/21/24, 9:46 PM

if you're concerned about routing/performance

B

bunOP•3/22/24, 4:14 AM

u cant use cloudflare cdn cache for R2, so no matter what you get charged for GetObject operations, unless you use cache api specifically, which just caches stuff in specific datacenters

Bbun u cant use cloudflare cdn cache for R2, so no matter what you get charged for Ge...

K

Karew•3/22/24, 5:06 AM

No, you can use the regular Cloudflare CDN with R2 if you connect a custom domain. If you get a

CF-Cache-Status: HIT

CF-Cache-Status: HIT

CF-Cache-Status: HIT

CF-Cache-Status: HIT, there is no charge on R2 at all, no request is made to the R2 backend if the Cloudflare cache serves it.
You're correct about this behavior only being per-datacenter though, so a quiet datacenter will not have cache hits nearly as often.

M

mickiewicz12•3/22/24, 6:23 AM

Hi,

I have an application for generating images connected with R2.
I noticed that some of the output links with images are delayed up to several minutes, until then, a 404 page is displayed
I have my own domain connected and all rate limits and protections for this bucket are disabled in the WAF tab.
Users generate about 3 images/second. Is it possible that this is too much for R2 or am I missing something?
I also used AWS S3 and Google storage and did not observe a similar experience even under much higher loads.

Thanks in advance.

P

paulrogers_sc•3/22/24, 1:21 PM

Greetings, I'm running into trouble trying to remove DNS for a previously R2-connected-domain, yet the bucket has been deleted.
So trying to click through this Domain Access link just produces "The specified bucket does not exist. (Code: 10006) " and a partially rendered bucket page.
https://dash.cloudflare.com/b5317876d8ea502bb62c977b940cab89/r2/default/buckets/sc-user-content/settings

Recreating the bucket is also blocked. What's the next step to get this resolved?

H

heyjay•3/22/24, 1:37 PM

I am using a CDN that uses R2. I upload to it using the app called Forklift. The CDN asset produces a 404 error unless I add a query param.

curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1.mp4\?hi

curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1.mp4\?hi

curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1.mp4\?hi

curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1.mp4\?hi will give me a 200 response but

curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1

curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1

curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1

curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1 will give me a 404 response

It's also only

*.mp4

*.mp4

*.mp4

*.mp4 files this happens with. My

*.png

*.png

*.png

*.png files work great

Hheyjay I am using a CDN that uses R2. I upload to it using the app called Forklift. The...

C

Chaika•3/22/24, 1:40 PM

Your 404s are being cached, and also

curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1

will never match because it's missing the .mp4 extension entirely, which is part of the object key, should be

https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1.mp4

https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1.mp4

https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1.mp4

https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1.mp4

H

heyjay•3/22/24, 1:40 PM

Yeah sorry... deleted that...

H

heyjay•3/22/24, 1:41 PM

curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1.mp4

curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1.mp4

curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1.mp4

curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1.mp4

H

heyjay•3/22/24, 1:41 PM

It gives me a 404

H

heyjay•3/22/24, 1:41 PM

❯ curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1.mp4
HTTP/2 404 
date: Fri, 22 Mar 2024 13:40:53 GMT
content-type: text/html
vary: Accept-Encoding
cache-control: max-age=14400
cf-cache-status: HIT
age: 562992
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=VjFNvapiIcOqkZjiRbA%2FM1%2BbpnwVbcLQzgwoFfPSqCo%2Bq8Ex0FFUll2jk5qAMb0SeVp%2FDBlR4EMibQ%2FdODzzVa%2FdNa9UaTA2QV7mep9U0wob9fYUgkU14CUYjyeunZkJ5rtWoN%2FFYDCtO9rYNA%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8686989d2d358d2c-DFW
alt-svc: h3=":443"; ma=86400

❯ curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1.mp4
HTTP/2 404 
date: Fri, 22 Mar 2024 13:40:53 GMT
content-type: text/html
vary: Accept-Encoding
cache-control: max-age=14400
cf-cache-status: HIT
age: 562992
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=VjFNvapiIcOqkZjiRbA%2FM1%2BbpnwVbcLQzgwoFfPSqCo%2Bq8Ex0FFUll2jk5qAMb0SeVp%2FDBlR4EMibQ%2FdODzzVa%2FdNa9UaTA2QV7mep9U0wob9fYUgkU14CUYjyeunZkJ5rtWoN%2FFYDCtO9rYNA%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8686989d2d358d2c-DFW
alt-svc: h3=":443"; ma=86400

❯ curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1.mp4
HTTP/2 404 
date: Fri, 22 Mar 2024 13:40:53 GMT
content-type: text/html
vary: Accept-Encoding
cache-control: max-age=14400
cf-cache-status: HIT
age: 562992
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=VjFNvapiIcOqkZjiRbA%2FM1%2BbpnwVbcLQzgwoFfPSqCo%2Bq8Ex0FFUll2jk5qAMb0SeVp%2FDBlR4EMibQ%2FdODzzVa%2FdNa9UaTA2QV7mep9U0wob9fYUgkU14CUYjyeunZkJ5rtWoN%2FFYDCtO9rYNA%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8686989d2d358d2c-DFW
alt-svc: h3=":443"; ma=86400

❯ curl -I https://cdn.fullqueuedeveloper.com/S5/S5E9-video-1.mp4
HTTP/2 404 
date: Fri, 22 Mar 2024 13:40:53 GMT
content-type: text/html
vary: Accept-Encoding
cache-control: max-age=14400
cf-cache-status: HIT
age: 562992
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=VjFNvapiIcOqkZjiRbA%2FM1%2BbpnwVbcLQzgwoFfPSqCo%2Bq8Ex0FFUll2jk5qAMb0SeVp%2FDBlR4EMibQ%2FdODzzVa%2FdNa9UaTA2QV7mep9U0wob9fYUgkU14CUYjyeunZkJ5rtWoN%2FFYDCtO9rYNA%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8686989d2d358d2c-DFW
alt-svc: h3=":443"; ma=86400

C

Chaika•3/22/24, 1:41 PM

see the cache there?

C

Chaika•3/22/24, 1:42 PM

You're caching 404s. You can use a Cache Rule to override the status by status code. Either to shorter, or no cache at all, or manually call purge whenever you call an upload, ex:

H

heyjay•3/22/24, 1:43 PM

Ohhhh that's good to know!! Thank you so much! This is my first time dealing with R2s and this isn't even mine. I'm working for someone else.

H

heyjay•3/22/24, 1:43 PM

I'll see if he can change the cache rule

C

Chaika•3/22/24, 1:46 PM

Cache Rules are under: Caching -> Cache Rules
https://dash.cloudflare.com/?to=/:account/:zone/caching/cache-rules/new

Easiest way to match it would be

Hostname

Hostname

Hostname

Hostname

equals

equals

equals

equals

cdn.fullqueuedeveloper.com

cdn.fullqueuedeveloper.com

cdn.fullqueuedeveloper.com

cdn.fullqueuedeveloper.com

He's already got an existing Page Rule or Cache Rule for it to cache it that long. If it's a Page Rule, he should recreate it as a Cache Rule. If it's a Cache Rule, he should just modify the existing.

The exact TTL is up to you. Cloudflare caches 404s for 3 minutes by default for example.

M

mickiewicz12•3/22/24, 2:22 PM

Can my problem be also related to 404s caching?

CChaika Cache Rules are under: Caching -> Cache Rules https://dash.cloudflare.com/?to=/:...

H

heyjay•3/22/24, 2:31 PM

Thanks!!

He was able to update the rules and 404s are no longer caching... Now I just need to figure out why content Types are wrong... Uploading a .mp4 file does not give me a content type of

vide/mp4

vide/mp4

vide/mp4

vide/mp4 but maybe that's also a caching error?

Hheyjay Thanks!! He was able to update the rules and 404s are no longer caching... Now ...

I

Isaac McFadyen•3/22/24, 2:35 PM

When you upload a file to R2 you need to specify the content type yourself.

I

Isaac McFadyen•3/22/24, 2:36 PM

It's not auto-detected. I believe it defaults to

binary/octet-stream

binary/octet-stream

binary/octet-stream

binary/octet-stream.

H

heyjay•3/22/24, 2:37 PM

Oh that’s really good to know!

H

heyjay•3/22/24, 2:37 PM

Maybe there’s some settings I need to set in my client to do that for me.

H

heyjay•3/22/24, 2:37 PM

I appreciate the help!

M

maco6517•3/22/24, 2:58 PM

multi-thread copy: failed to write chunk: failed to upload chunk 4 with 5242880 bytes: NoSuchUpload: The specified multipart upload does not exist

M

maco6517•3/22/24, 2:59 PM

I get a rclone error when uploading a file larger than 300mb

Mmaco6517 multi-thread copy: failed to write chunk: failed to upload chunk 4 with 5242880 ...

K

Karew•3/22/24, 7:52 PM

Are you specifically using the R2 config mode on Rclone? Make sure your Rclone is the newest version and set up your connection again, Rclone has a specific choice for R2 connections when setting one up

S

sha386•3/24/24, 1:53 AM

Is it possible to set a key for example im uploading a png file with the key as "textures/blocks/amethyst_cluster.png"

What issues did you have with b2 for hot storage

Similar Threads

Similar Threads

Similar Threads