How to cache googlebot requests? - Cloudflare Developers

© 2026 Hedgehog Software, LLC

More

Communities Docs About Terms Privacy

maco6517

M

maco6517OP•11/26/24, 3:03 AM

helpme

Unsmart

U

Unsmart•11/26/24, 3:14 AM

I make a lot of requests to R2 from every single CF region but getting a definitive answer on which data center is the one hosting the bucket isnt really possible. It varies which dc gives the lowest latency all the time all I could say is east is in the east and west is in the west for NA for example lol

Gezim

G

Gezim•11/26/24, 3:29 AM

Hi folks. I'm getting 502 errors much more than I would like. Is this normal?

maco6517

Mmaco6517 How to cache googlebot requests?

Isaac McFadyen

I

Isaac McFadyen•11/26/24, 3:40 AM

?crossposting

Flare

F

Flare•11/26/24, 3:40 AM

Please do not post your question in multiple channels/post it multiple times per the rules at ⁠#

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

Gezim

GGezim Hi folks. I'm getting 502 errors much more than I would like. Is this normal?

Gezim

G

Gezim•11/26/24, 5:05 AM

Could my 502s be caused because I'm throttling my network? Is cloudflare using workers internally to for

put_object

put_object

and the worker is getting killed?

Pinky Kyulkyung

P

Pinky Kyulkyung•11/26/24, 5:24 AM

hello every one!
i have a question that i add a member to join my account but he cant access my R2 Storage, when he click it shows that

Pinky Kyulkyung

P

Pinky Kyulkyung•11/26/24, 5:25 AM

alex (he/him)

A

alex (he/him)•11/26/24, 6:37 AM

Is there a overview available which cities/locations are mapped to which location hint? I need to figure out the difference between Western Europe and Eastern Europe

daniel-supalang

D

daniel-supalang•11/26/24, 9:28 AM

Hi.

If I use Cloudflare R2 with a CDN, do GET requests (Class B Operations) count when it fetches data from CDN?

Original message was deleted

daniel-supalang

D

daniel-supalang•11/26/24, 9:43 AM

Thank you so much! Then, if I set the time-to-live (TTL) to a really long duration (e.g., 3 months) with Cache Rules, does that mean Class B operations won't be counted during that period?

Original message was deleted

daniel-supalang

D

daniel-supalang•11/26/24, 9:47 AM

Thank you so much!

101budspencer

1

101budspencer•11/26/24, 4:41 PM

What was that again in Cloudflare, where you can restrict access to bucket files URL's without losing cache features?

101budspencer

1

101budspencer•11/26/24, 4:42 PM

Somehow generating some key and appending it in the URL when accessing it and if the key is not valid it blocks you

101budspencer

1101budspencer What was that again in Cloudflare, where you can restrict access to bucket files...

digitalpoint

D

digitalpoint•11/26/24, 4:45 PM

Are you talking about the is_timed_hmac_valid_v0() function?

https://developers.cloudflare.com/ruleset-engine/rules-language/functions/#hmac-validation

Cloudflare Docs

Functions | Cloudflare Ruleset Engine docs

The Cloudflare Rules language provides functions for manipulating and validating values in an expression:

101budspencer

1

101budspencer•11/26/24, 4:45 PM

Yes

101budspencer

1

101budspencer•11/26/24, 4:45 PM

Thanks!

101budspencer

1

101budspencer•11/26/24, 4:45 PM

https://tenor.com/view/thank-you-baby-snoop-dogg-cameo-thanks-thanks-love-gif-18006774

Tenor

digitalpoint

D

digitalpoint•11/26/24, 4:48 PM

Haven't tried using it with cache, but you would probably need to use the "ignore query string" stuff for the cache level. But also, if you did that, it kind of defeats the purpose of the hmac validation since hitting the cache wouldn't actually be doing the validation.

So not really sure it would actually work (both hmac validation and cache hit).

101budspencer

1

101budspencer•11/26/24, 4:50 PM

Worker runs no matter what I believe, maybe using that instead

digitalpoint

D

digitalpoint•11/26/24, 4:50 PM

Or wait... maybe it would work since I think the Page Rules are before the CDN cache in the traffic sequence

101budspencer

1

101budspencer•11/26/24, 4:50 PM

Or that

digitalpoint

D

digitalpoint•11/26/24, 5:04 PM

That reminds me... I'm using

is_timed_hmac_valid_v0()

is_timed_hmac_valid_v0()

in some places instead of presigned URLs because presigned URLs don't support HTTP/2 or HTTP/3.

If someone is reading things, can we finally get presigned URLs on a protocol that came out more recently than almost 30 years ago.

digitalpoint

Ddigitalpoint Haven't tried using it with cache, but you would probably need to use the "ignor...

Chaika

C

Chaika•11/26/24, 6:06 PM

fw/waf runs before cache/origin

tezcan

T

tezcan•11/26/24, 6:53 PM

hi, I'm trying to upload a file to my r2 bucket with rclone but I get this error. "S3: PutObject, exceeded maximum number of attempts" it tries it twice and uploads successfully on the 3rd attempt but it costs me time. I upload a maximum of 10 new files per day so there's no way I can exceed any limit.

tezcan

T

tezcan•11/26/24, 6:58 PM

This is the command I use "rclone sync /remote r2:bucket --progress --transfers=8 --s3-upload-concurrency=8 --s3-upload-cutoff=512M --s3-chunk-size=512M --fast-list --size-only " There was no problem for a week, but today it started giving this error.

tezcan

T

tezcan•11/26/24, 6:59 PM

most files +400mb

tezcan

T

tezcan•11/26/24, 7:01 PM

Let me try.

tezcan

T

tezcan•11/26/24, 7:58 PM

When I looked at the log file in more detail, I saw this "can't copy - source file is being updated". I am connected to the remote server with samba/cifs, I guess it is a problem related to that. The file was actually edited 1 hour ago, but rclone still sees it as being edited. But it doesn't matter. --local-no-check-updated This command solved the problem. Sorry for wasting your time. thanks.

Original message was deleted

bright_parrot_49151

B

bright_parrot_49151•11/27/24, 11:13 AM

Advany

A

Advany•11/27/24, 11:23 AM

I am getting errors for writing and reading to R2 from a workers. I keep getting:

put: We encountered an internal error. Please try again. (10001)

Writing 1 or 2 things per minute orso

Objects are around 200kb orso text

Keys are something like "domain.com/page#target"

am I doing something wrong?

This is really making it difficult to work with R2

dave

Ddave I'm running into a fair amount of these errors: ✘ [ERROR] Error: put: We enco...

Advany

A

Advany•11/27/24, 11:25 AM

where you able to solve this?

Original message was deleted

Advany

A

Advany•11/27/24, 12:00 PM

this one is not doing that. I am thinking of just using the S3 api. Hopefully that is more stable?

Original message was deleted

Advany

A

Advany•11/27/24, 12:11 PM

only doing like 2-3 writes and reads per minute. Different keys.

Original message was deleted

Advany

A

Advany•11/27/24, 12:20 PM

I think so?

Screenshot_2024-11-27_at_13.20.09.png

Screenshot_2024-11-27_at_13.19.57.png

Advany

AAdvany I think so?

Advany

A

Advany•11/27/24, 12:21 PM

maybe a bit more reads that are also happening in a other worker

Original message was deleted

Advany

A

Advany•11/27/24, 1:02 PM

nope... really dont get the error

Advany

AAdvany nope... really dont get the error

Advany

A

Advany•11/27/24, 1:03 PM

strangly... I have the same script, same setup, in a different account. No issues...

Advany

AAdvany strangly... I have the same script, same setup, in a different account. No issue...

Advany

A

Advany•11/27/24, 1:08 PM

with higher write/reads

Screenshot_2024-11-27_at_14.08.22.png

Screenshot_2024-11-27_at_14.08.16.png

Advany

AAdvany where you able to solve this?

dave

D

dave•11/27/24, 1:11 PM

No. We just retry using aws4fetch if the data really matters. We only use the bindings if it’s a quick script that isn’t important

dave

Ddave No. We just retry using aws4fetch if the data really matters. We only use the bi...

Advany

A

Advany•11/27/24, 3:10 PM

Thanks. Will do the same

Strange enough same worker, more traffic working fine in one account and giving errors in the other one

Advany

AAdvany Thanks. Will do the same Strange enough same worker, more traffic working fine ...

dave

D

dave•11/27/24, 3:33 PM

Yeah, I didn't see much of a pattern either.

para

P

para•11/27/24, 5:15 PM

Hi, we're switching from S3 to R2 and have about 300k original images. On top of this, we also have thumbnail versions stored so 600k images in total. Does it make economical sense to store both versions with R2 compared to using transforms from the original image? The thought process is that the monthly storage would cost less than the possible transforms.

maurus

M

maurus•11/27/24, 5:40 PM

Hi, I'm trying to debug CORS on R2. I've configured a CORS policy on my R2 bucket in the Cloudflare dashboard, but when doing a preflight request I'm getting 403

<?xml version="1.0" encoding="UTF-8"?><Error><Code>Unauthorized</Code><Message>CORS not configured for this bucket</Message></Error>

<?xml version="1.0" encoding="UTF-8"?><Error><Code>Unauthorized</Code><Message>CORS not configured for this bucket</Message></Error>

(I'm using presigned URLs)

Does it take some time until CORS policy changes are propagated?

Original message was deleted

para

P

para•11/27/24, 6:44 PM

Image size average is 150kb, so should be around ~100 GB.

para

P

para•11/27/24, 6:48 PM

Perfect, makes this much easier. Thanks for your insight.

Original message was deleted

Advany

A

Advany•11/27/24, 9:14 PM

You are really awesome with all the help you give people with r2

prstsn

P

prstsn•11/27/24, 10:35 PM

Hello, i'm new here.
I have trouble with uploading files in R2 with correct key.
When i try to generate upload presign url (that i'm using in browser to upload file) with getSignedUrl it appends bucket name in the end of the key automatically so it creates folder with key i passed and makes file in it with bucket name. Key looks something like this: avatar.png/bucketname instead of: avatar.png.
If i directly upload with putObject from node, it uploads file with correct key.
So only generating presignUrl has problem.
I have forcePathStyle: true in my s3client config.

Has anyone ever had problem like this?

imag

I

imag•11/27/24, 10:50 PM

Hey guys a bit of an odd question but is there any way to get an objects size (content length if thats what you'd call it) within a cf worker? I see that the aws-sdk's might let you do it but Im looking towards handling it inside a worker binding specifically, anyone come across this before?

roter_schnee

R

roter_schnee•11/28/24, 1:25 AM

Hey, guys, newbie here.
I am going to publish a site with cf pages.
I do not expect to get a lot of traffic (total monthly visitors under 3-4k), 90% of traffic is expected to come from one region (Eastern Eu).
The site will be serving some static images (no image transformations, 40-60 images total, 2-3mb each).
So I am not sure which approach fits better described case. R2 with custom domain or CF Images? Would appreciate any advice.