@Chaika i honestly would prefer to do this with r2 + cache. but im not sure how the cache thing work

Bbun @Chaika i honestly would prefer to do this with r2 + cache. but im not sure how ...

C

Chaika•2/2/24, 12:58 AM

Sounds like you were trying to use Workers? With R2, you would be best off with just an R2 Custom Domain and a Cache rule to force it

B

bunOP•2/2/24, 12:58 AM

yeah i have a custom r2 domain, but when doing bindings is that domain used ?

C

Chaika•2/2/24, 12:58 AM

If you use an R2 Custom Domain then you wouldn't pay for Worker Invocations and any cache hits are free

Bbun yeah i have a custom r2 domain, but when doing bindings is that domain used ?

C

Chaika•2/2/24, 12:58 AM

what do you mean "when doing bindings"?

B

bunOP•2/2/24, 12:59 AM

i wasnt accessing r2 storage through fetch on my custom domain. i was accessing them through

wrangler.toml

wrangler.toml

storage bindings

C

Chaika•2/2/24, 1:00 AM

That wouldn't use the Custom domain or cache by default, you'd have to use the cache api yourself

C

Chaika•2/2/24, 1:01 AM

i couldnt get it to work, i would get cache miss everytime

The Cache API doesn't work/will always return miss in a bunch of environments, including dev

However, any Cache API operations in the Cloudflare Workers dashboard editor, Playground previews, and any *.workers.dev deployments will have no impact. For Workers fronted by Cloudflare Access
, the Cache API is not currently available. Only Workers deployed to custom domains have access to functional cache operations.

https://developers.cloudflare.com/workers/runtime-apis/cache/

B

bunOP•2/2/24, 1:01 AM

yeah i was doing the cache api manually but it wasnt working for some reason

B

bunOP•2/2/24, 1:01 AM

yeah my workers were also deployed with custom domains

B

bunOP•2/2/24, 1:01 AM

i can show you my code if that will help

C

Chaika•2/2/24, 1:01 AM

Sure

B

bunOP•2/2/24, 1:02 AM

app.use("*", async (c, next) => {
  const t = performance.now();
  const cache = caches.default;

  const cacheUrl = new URL(c.req.url);

  const cacheKey = new Request(cacheUrl.toString(), c.req);

  let response = await cache.match(cacheKey);

  if (response) {
    c.env.myData = { cachedData: await response.json() };
    console.log("cache hit", performance.now() - t);
    return next();
  }

  const object = await c.env.MY_R2_STORAGE.get("some-data.json");

  if (!object) {
    throw new Error("Data not found");
  }

  const headers = new Headers();

  object.writeHttpMetadata(headers);
  headers.set("etag", object.httpEtag);
  const status = object.body ? 200 : 304;

  const [body1, body2] = object.body.tee();

  response = new Response(body1, {
    headers,
    status,
  });

  response.headers.append("Cache-Control", "s-maxage=31536000");

  c.executionCtx.waitUntil(cache.put(cacheKey, response));

  if (!cachedData) {
    throw new Error("Data not found");
  }

  c.env.myData = { cachedData: await streamToJson(body2) };

  console.log("cache miss", performance.now() - t);

  return next();
});

app.use("*", async (c, next) => {
  const t = performance.now();
  const cache = caches.default;

  const cacheUrl = new URL(c.req.url);

  const cacheKey = new Request(cacheUrl.toString(), c.req);

  let response = await cache.match(cacheKey);

  if (response) {
    c.env.myData = { cachedData: await response.json() };
    console.log("cache hit", performance.now() - t);
    return next();
  }

  const object = await c.env.MY_R2_STORAGE.get("some-data.json");

  if (!object) {
    throw new Error("Data not found");
  }

  const headers = new Headers();

  object.writeHttpMetadata(headers);
  headers.set("etag", object.httpEtag);
  const status = object.body ? 200 : 304;

  const [body1, body2] = object.body.tee();

  response = new Response(body1, {
    headers,
    status,
  });

  response.headers.append("Cache-Control", "s-maxage=31536000");

  c.executionCtx.waitUntil(cache.put(cacheKey, response));

  if (!cachedData) {
    throw new Error("Data not found");
  }

  c.env.myData = { cachedData: await streamToJson(body2) };

  console.log("cache miss", performance.now() - t);

  return next();
});

app.use("*", async (c, next) => {
  const t = performance.now();
  const cache = caches.default;

  const cacheUrl = new URL(c.req.url);

  const cacheKey = new Request(cacheUrl.toString(), c.req);

  let response = await cache.match(cacheKey);

  if (response) {
    c.env.myData = { cachedData: await response.json() };
    console.log("cache hit", performance.now() - t);
    return next();
  }

  const object = await c.env.MY_R2_STORAGE.get("some-data.json");

  if (!object) {
    throw new Error("Data not found");
  }

  const headers = new Headers();

  object.writeHttpMetadata(headers);
  headers.set("etag", object.httpEtag);
  const status = object.body ? 200 : 304;

  const [body1, body2] = object.body.tee();

  response = new Response(body1, {
    headers,
    status,
  });

  response.headers.append("Cache-Control", "s-maxage=31536000");

  c.executionCtx.waitUntil(cache.put(cacheKey, response));

  if (!cachedData) {
    throw new Error("Data not found");
  }

  c.env.myData = { cachedData: await streamToJson(body2) };

  console.log("cache miss", performance.now() - t);

  return next();
});

app.use("*", async (c, next) => {
  const t = performance.now();
  const cache = caches.default;

  const cacheUrl = new URL(c.req.url);

  const cacheKey = new Request(cacheUrl.toString(), c.req);

  let response = await cache.match(cacheKey);

  if (response) {
    c.env.myData = { cachedData: await response.json() };
    console.log("cache hit", performance.now() - t);
    return next();
  }

  const object = await c.env.MY_R2_STORAGE.get("some-data.json");

  if (!object) {
    throw new Error("Data not found");
  }

  const headers = new Headers();

  object.writeHttpMetadata(headers);
  headers.set("etag", object.httpEtag);
  const status = object.body ? 200 : 304;

  const [body1, body2] = object.body.tee();

  response = new Response(body1, {
    headers,
    status,
  });

  response.headers.append("Cache-Control", "s-maxage=31536000");

  c.executionCtx.waitUntil(cache.put(cacheKey, response));

  if (!cachedData) {
    throw new Error("Data not found");
  }

  c.env.myData = { cachedData: await streamToJson(body2) };

  console.log("cache miss", performance.now() - t);

  return next();
});

B

bunOP•2/2/24, 1:04 AM

as you can see i was using same

cacheKey

cacheKey

cacheKey

cacheKey for both

cache.match

cache.match

cache.match

cache.match and

cache.put

cache.put

. however it was always a miss

B

bunOP•2/2/24, 1:05 AM

im super confused on why

Request

Request

constructed from

request.url

request.url

request.url

request.url has to be the cache key

C

Chaika•2/2/24, 1:08 AM

Its built on top of the normal cdn cache

C

Chaika•2/2/24, 1:09 AM

it could be a valid url that isn't in your zone as long as it's not another cf website, some people use example.invalid or stuf like that

C

Chaika•2/2/24, 1:09 AM

if you pass it a request with headers though it respects some of them: https://developers.cloudflare.com/workers/runtime-apis/cache/#parameters-1

Cache · Cloudflare Workers docs

Control reading and writing from the Cloudflare global network cache.

C

Chaika•2/2/24, 1:21 AM

I don't see anything wrong with that code though as long as it is running on a custom domain/route on your own domain (i.e not the workers.dev, not behind access.
It just never returns hit on your end?
I would question though if it's the same response json, why are you having it differ by url?

B

bunOP•2/2/24, 1:24 AM

differ by url? you mean

const cacheUrl = new URL(c.req.url)

const cacheUrl = new URL(c.req.url)

const cacheUrl = new URL(c.req.url)

const cacheUrl = new URL(c.req.url) ?

B

bunOP•2/2/24, 1:24 AM

yeah i was just super confused by this methodology of caching something so while trying to make it work i just didnt question it and made it as close to the documentation example as possible

B

bunOP•2/2/24, 1:25 AM

const request = await fetch("https://r2.sunney.dev/data.json", {
    cf: {
      cacheTtl: 60 * 60 * 24,
      cacheEverything: true,
      cacheKey: someCustomKey,
    },
  }).then((res) => res.json());

const request = await fetch("https://r2.sunney.dev/data.json", {
    cf: {
      cacheTtl: 60 * 60 * 24,
      cacheEverything: true,
      cacheKey: someCustomKey,
    },
  }).then((res) => res.json());

const request = await fetch("https://r2.sunney.dev/data.json", {
    cf: {
      cacheTtl: 60 * 60 * 24,
      cacheEverything: true,
      cacheKey: someCustomKey,
    },
  }).then((res) => res.json());

const request = await fetch("https://r2.sunney.dev/data.json", {
    cf: {
      cacheTtl: 60 * 60 * 24,
      cacheEverything: true,
      cacheKey: someCustomKey,
    },
  }).then((res) => res.json());

this works well enough

C

Chaika•2/2/24, 1:27 AM

same cache anyway, that'll work. Embedding it in a Worker isn't a bad idea either, I do that for html just for fun & speed and it works well, super fast

B

bunOP•2/2/24, 1:27 AM

thanks for all the help

D

dave•2/2/24, 5:05 AM

if I wanna gzip things before putting them into R2, do I really have to await on the response?

D

dave•2/2/24, 5:06 AM

if I don't, it complains that the content length is not known

R

Rahul Raikwar•2/2/24, 12:12 PM

How can achieve this - setting up a webhook in next js project to automatically trigger builds on Cloudflare when changes shopify admin products, collections, blog articles, and pages.

Original message was deleted

I

Isaac McFadyen•2/2/24, 1:15 PM

Yes.

I

Isaac McFadyen•2/2/24, 1:16 PM

Not sure if there's a publicly stated reason but I'd just consider the fact that some Workers customers get millions of requests per second - at that scale, decompressing the Worker itself isn't an insignificant cost and I assume they preferred decompression speed over size.

I

Isaac McFadyen•2/2/24, 1:17 PM

It would also be a fairly major task to switch over to a new algorithm considering they'd need to decompress and recompress thousands of old Workers and switch all of the runtime components to the new algorithm across thousands of servers, all without any downtime.

Ppeshakoo Hey Dave! if you don't mind could you show me how you did this?? I'm doing somet...

D

dave•2/2/24, 2:12 PM

Hmm lemme check if we actually ended up doing it or not. ;p

Original message was deleted

D

dave•2/2/24, 2:14 PM

Likely they use Workers for some of the Workers supporting infrastructure. Workers only support compressing gzip streams. https://developer.mozilla.org/en-US/docs/Web/API/CompressionStream/CompressionStream

Ppeshakoo Hey Dave! if you don't mind could you show me how you did this?? I'm doing somet...

D

dave•2/2/24, 2:19 PM

ah sorry, we ended up not needing to use ListObjects. That said, I think it should be pretty straightforward, would be a cool blog post if you end up doing it

Original message was deleted

D

dave•2/2/24, 2:20 PM

Yes, but maintaining an-out-of-tree patch to V8 and making a non-standard Web API kinda sucks

Original message was deleted

D

dave•2/2/24, 2:25 PM

pretty sure they did a blog post on it

Original message was deleted

D

dave•2/2/24, 2:28 PM

https://blog.cloudflare.com/deploying-firmware-at-cloudflare-scale-how-we-update-thousands-of-servers-in-more-than-285-cities

Original message was deleted

D

dave•2/2/24, 2:28 PM

https://blog.cloudflare.com/anchoring-trust-a-hardware-secure-boot-story

The Cloudflare Blog

Anchoring Trust: A Hardware Secure Boot Story

As a security company, we pride ourselves on finding innovative ways to protect our platform to, in turn, protect the data of our customers. Part of this approach is implementing progressive methods in protecting our hardware at scale.

Original message was deleted

C

Chaika•2/2/24, 2:29 PM

They're just Linux machines. Their main inbound proxy is just nginx (modified though of course), they made some of their patches public. Workers use workerd to run (https://github.com/cloudflare/workerd), built on top of Chromium's V8 Javascript Engine, nothing too crazy

C

Chaika•2/2/24, 2:29 PM

they go over a lot of it in their blogs like the ones linked above

C

Chaika•2/2/24, 2:29 PM

also Cloudflare lol

M

Mehanika•2/2/24, 2:34 PM

Anyone successfully integrated Sentry and their Worker? I added the projects but they dont show up on the list for reasons unknown