And I was thinking cloudflare workers would work because of the low latency overall.

C

Chaika•2/1/24, 6:54 PM

udp sockets are somewhere in the plans, one day hopefully

SScai And I was thinking cloudflare workers would work because of the low latency over...

C

Chaika•2/1/24, 6:54 PM

by default it would be closest to the visitor/executor of the worker, not the server

G

glamboyosa•2/1/24, 7:09 PM

heya guys. possible noob question as I'm super new to CF using it for a side project (https://vaella.glamboyosa.xyz) and environment variables and CF workers but.

say I have an environment variable in my dev.vars called database_id

and in the [vars] I reference database_id as database_id=$database_id

and also for environment.development

and it all works fine.

but how would I reference environment variables for production that way?

I am aware you could set up a GitHub action but I don't want to or need to so early. and I'd like to commit my wrangler.toml file with my sensitive data secure.

Vaella

Private Audio Rant Rooms

G

glamboyosa•2/1/24, 7:09 PM

so kinda like on the Vercel dashboard where you can specify environment variables and it'll be read for prod deploys.

G

glamboyosa•2/1/24, 7:09 PM

like I wanna be able to tell D1 to reference the database_id variable and it kinda point to the var in prod somehow when I use wrangler.

same for using wrangler deploy ...

I'd appreciate any guidance or examples (so j don't waste anyone's time if it's a bit too noob-ish!)

A

AKStriker98•2/1/24, 7:32 PM

Would a worker with service bindings to itself to boost the number of sub requests for logging purposes start interrupting the simultaneous open requests unless an event.waitUntil() and return is coded?

D

dave•2/1/24, 8:31 PM

huh, when my Worker does a

fetch

fetch

to a URL outside of my zone, polish is used...?

Original message was deleted

R

Radoš•2/1/24, 10:35 PM

1) Create

package.json

package.json

file. The main thing you care about is:

...
"type": "module",
"workspaces": [
  "projects/*"
],
"scripts": {
  "stripe-worker": "cd projects/stripe-worker && bun",
  "main-site": "cd projects/main-site && bun",
  "mail-worker": "cd projects/mail-worker && bun"
},
...

...
"type": "module",
"workspaces": [
  "projects/*"
],
"scripts": {
  "stripe-worker": "cd projects/stripe-worker && bun",
  "main-site": "cd projects/main-site && bun",
  "mail-worker": "cd projects/mail-worker && bun"
},
...

...
"type": "module",
"workspaces": [
  "projects/*"
],
"scripts": {
  "stripe-worker": "cd projects/stripe-worker && bun",
  "main-site": "cd projects/main-site && bun",
  "mail-worker": "cd projects/mail-worker && bun"
},
...

...
"type": "module",
"workspaces": [
  "projects/*"
],
"scripts": {
  "stripe-worker": "cd projects/stripe-worker && bun",
  "main-site": "cd projects/main-site && bun",
  "mail-worker": "cd projects/mail-worker && bun"
},
...

2) Create a projects dir in the root with desired project folders. Each folder will have whatever code you need
3) Adjust

scripts

scripts

. In my case I use bun as why not. No need for pnpm workspace as this works the same if not better.
4) Terminal

bun main-site install

bun main-site install

or

bun install

bun install

depends on what you want to do.
5) Adjust

scripts

scripts

in

package.json

package.json

so you can do something like

bun main-site deploy

bun main-site deploy

or

bun mail-worker dev

bun mail-worker dev

No need to overcomplicate this. Use common libs in the root

package.json

package.json

and let bun to handle the rest.

K

kayo•2/1/24, 11:12 PM

Hey, is Workers a SaaS thing or a FaaS?

C

Chaika•2/1/24, 11:27 PM

FaaS

J

Justintime•2/1/24, 11:28 PM

does anyone know a workaround for the following

workers.api.error.malformed_param: must specify either 'script' or 'service' [code: 10004]

workers.api.error.malformed_param: must specify either 'script' or 'service' [code: 10004]

B

bun•2/2/24, 12:57 AM

@Chaika i honestly would prefer to do this with r2 + cache. but im not sure how the cache thing works exactly? i get that cloudflare has some cache but how accessible is it to workers? and most importantly whats with the weird cache accessing syntax? i couldnt get it to work, i would get cache miss everytime

Bbun @Chaika i honestly would prefer to do this with r2 + cache. but im not sure how ...

C

Chaika•2/2/24, 12:58 AM

Sounds like you were trying to use Workers? With R2, you would be best off with just an R2 Custom Domain and a Cache rule to force it

B

bun•2/2/24, 12:58 AM

yeah i have a custom r2 domain, but when doing bindings is that domain used ?

C

Chaika•2/2/24, 12:58 AM

If you use an R2 Custom Domain then you wouldn't pay for Worker Invocations and any cache hits are free

Bbun yeah i have a custom r2 domain, but when doing bindings is that domain used ?

C

Chaika•2/2/24, 12:58 AM

what do you mean "when doing bindings"?

B

bun•2/2/24, 12:59 AM

i wasnt accessing r2 storage through fetch on my custom domain. i was accessing them through

wrangler.toml

wrangler.toml

storage bindings

C

Chaika•2/2/24, 1:00 AM

That wouldn't use the Custom domain or cache by default, you'd have to use the cache api yourself

C

Chaika•2/2/24, 1:01 AM

i couldnt get it to work, i would get cache miss everytime

The Cache API doesn't work/will always return miss in a bunch of environments, including dev

However, any Cache API operations in the Cloudflare Workers dashboard editor, Playground previews, and any *.workers.dev deployments will have no impact. For Workers fronted by Cloudflare Access
, the Cache API is not currently available. Only Workers deployed to custom domains have access to functional cache operations.

https://developers.cloudflare.com/workers/runtime-apis/cache/

B

bun•2/2/24, 1:01 AM

yeah i was doing the cache api manually but it wasnt working for some reason

B

bun•2/2/24, 1:01 AM

yeah my workers were also deployed with custom domains

B

bun•2/2/24, 1:01 AM

i can show you my code if that will help

C

Chaika•2/2/24, 1:01 AM

Sure

B

bun•2/2/24, 1:02 AM

app.use("*", async (c, next) => {
  const t = performance.now();
  const cache = caches.default;

  const cacheUrl = new URL(c.req.url);

  const cacheKey = new Request(cacheUrl.toString(), c.req);

  let response = await cache.match(cacheKey);

  if (response) {
    c.env.myData = { cachedData: await response.json() };
    console.log("cache hit", performance.now() - t);
    return next();
  }

  const object = await c.env.MY_R2_STORAGE.get("some-data.json");

  if (!object) {
    throw new Error("Data not found");
  }

  const headers = new Headers();

  object.writeHttpMetadata(headers);
  headers.set("etag", object.httpEtag);
  const status = object.body ? 200 : 304;

  const [body1, body2] = object.body.tee();

  response = new Response(body1, {
    headers,
    status,
  });

  response.headers.append("Cache-Control", "s-maxage=31536000");

  c.executionCtx.waitUntil(cache.put(cacheKey, response));

  if (!cachedData) {
    throw new Error("Data not found");
  }

  c.env.myData = { cachedData: await streamToJson(body2) };

  console.log("cache miss", performance.now() - t);

  return next();
});

app.use("*", async (c, next) => {
  const t = performance.now();
  const cache = caches.default;

  const cacheUrl = new URL(c.req.url);

  const cacheKey = new Request(cacheUrl.toString(), c.req);

  let response = await cache.match(cacheKey);

  if (response) {
    c.env.myData = { cachedData: await response.json() };
    console.log("cache hit", performance.now() - t);
    return next();
  }

  const object = await c.env.MY_R2_STORAGE.get("some-data.json");

  if (!object) {
    throw new Error("Data not found");
  }

  const headers = new Headers();

  object.writeHttpMetadata(headers);
  headers.set("etag", object.httpEtag);
  const status = object.body ? 200 : 304;

  const [body1, body2] = object.body.tee();

  response = new Response(body1, {
    headers,
    status,
  });

  response.headers.append("Cache-Control", "s-maxage=31536000");

  c.executionCtx.waitUntil(cache.put(cacheKey, response));

  if (!cachedData) {
    throw new Error("Data not found");
  }

  c.env.myData = { cachedData: await streamToJson(body2) };

  console.log("cache miss", performance.now() - t);

  return next();
});

app.use("*", async (c, next) => {
  const t = performance.now();
  const cache = caches.default;

  const cacheUrl = new URL(c.req.url);

  const cacheKey = new Request(cacheUrl.toString(), c.req);

  let response = await cache.match(cacheKey);

  if (response) {
    c.env.myData = { cachedData: await response.json() };
    console.log("cache hit", performance.now() - t);
    return next();
  }

  const object = await c.env.MY_R2_STORAGE.get("some-data.json");

  if (!object) {
    throw new Error("Data not found");
  }

  const headers = new Headers();

  object.writeHttpMetadata(headers);
  headers.set("etag", object.httpEtag);
  const status = object.body ? 200 : 304;

  const [body1, body2] = object.body.tee();

  response = new Response(body1, {
    headers,
    status,
  });

  response.headers.append("Cache-Control", "s-maxage=31536000");

  c.executionCtx.waitUntil(cache.put(cacheKey, response));

  if (!cachedData) {
    throw new Error("Data not found");
  }

  c.env.myData = { cachedData: await streamToJson(body2) };

  console.log("cache miss", performance.now() - t);

  return next();
});

app.use("*", async (c, next) => {
  const t = performance.now();
  const cache = caches.default;

  const cacheUrl = new URL(c.req.url);

  const cacheKey = new Request(cacheUrl.toString(), c.req);

  let response = await cache.match(cacheKey);

  if (response) {
    c.env.myData = { cachedData: await response.json() };
    console.log("cache hit", performance.now() - t);
    return next();
  }

  const object = await c.env.MY_R2_STORAGE.get("some-data.json");

  if (!object) {
    throw new Error("Data not found");
  }

  const headers = new Headers();

  object.writeHttpMetadata(headers);
  headers.set("etag", object.httpEtag);
  const status = object.body ? 200 : 304;

  const [body1, body2] = object.body.tee();

  response = new Response(body1, {
    headers,
    status,
  });

  response.headers.append("Cache-Control", "s-maxage=31536000");

  c.executionCtx.waitUntil(cache.put(cacheKey, response));

  if (!cachedData) {
    throw new Error("Data not found");
  }

  c.env.myData = { cachedData: await streamToJson(body2) };

  console.log("cache miss", performance.now() - t);

  return next();
});

B

bun•2/2/24, 1:04 AM

as you can see i was using same

cacheKey

cacheKey

for both

cache.match

cache.match

and

cache.put

cache.put

. however it was always a miss

B

bun•2/2/24, 1:05 AM

im super confused on why

Request

Request

constructed from

request.url

request.url

has to be the cache key

C

Chaika•2/2/24, 1:08 AM

Its built on top of the normal cdn cache

C

Chaika•2/2/24, 1:09 AM

it could be a valid url that isn't in your zone as long as it's not another cf website, some people use example.invalid or stuf like that

C

Chaika•2/2/24, 1:09 AM

if you pass it a request with headers though it respects some of them: https://developers.cloudflare.com/workers/runtime-apis/cache/#parameters-1

Cache · Cloudflare Workers docs

Control reading and writing from the Cloudflare global network cache.

C

Chaika•2/2/24, 1:21 AM

I don't see anything wrong with that code though as long as it is running on a custom domain/route on your own domain (i.e not the workers.dev, not behind access.
It just never returns hit on your end?
I would question though if it's the same response json, why are you having it differ by url?

B

bun•2/2/24, 1:24 AM

differ by url? you mean

const cacheUrl = new URL(c.req.url)

const cacheUrl = new URL(c.req.url)

const cacheUrl = new URL(c.req.url)

const cacheUrl = new URL(c.req.url) ?

B

bun•2/2/24, 1:24 AM

yeah i was just super confused by this methodology of caching something so while trying to make it work i just didnt question it and made it as close to the documentation example as possible

B

bun•2/2/24, 1:25 AM

const request = await fetch("https://r2.sunney.dev/data.json", {
    cf: {
      cacheTtl: 60 * 60 * 24,
      cacheEverything: true,
      cacheKey: someCustomKey,
    },
  }).then((res) => res.json());

const request = await fetch("https://r2.sunney.dev/data.json", {
    cf: {
      cacheTtl: 60 * 60 * 24,
      cacheEverything: true,
      cacheKey: someCustomKey,
    },
  }).then((res) => res.json());

const request = await fetch("https://r2.sunney.dev/data.json", {
    cf: {
      cacheTtl: 60 * 60 * 24,
      cacheEverything: true,
      cacheKey: someCustomKey,
    },
  }).then((res) => res.json());

const request = await fetch("https://r2.sunney.dev/data.json", {
    cf: {
      cacheTtl: 60 * 60 * 24,
      cacheEverything: true,
      cacheKey: someCustomKey,
    },
  }).then((res) => res.json());

this works well enough

C

Chaika•2/2/24, 1:27 AM

same cache anyway, that'll work. Embedding it in a Worker isn't a bad idea either, I do that for html just for fun & speed and it works well, super fast

B

bun•2/2/24, 1:27 AM

thanks for all the help

D

dave•2/2/24, 5:05 AM

if I wanna gzip things before putting them into R2, do I really have to await on the response?

D

dave•2/2/24, 5:06 AM

if I don't, it complains that the content length is not known

R

Rahul Raikwar•2/2/24, 12:12 PM

How can achieve this - setting up a webhook in next js project to automatically trigger builds on Cloudflare when changes shopify admin products, collections, blog articles, and pages.

Original message was deleted

I

Isaac McFadyen•2/2/24, 1:15 PM

Yes.

I

Isaac McFadyen•2/2/24, 1:16 PM

Not sure if there's a publicly stated reason but I'd just consider the fact that some Workers customers get millions of requests per second - at that scale, decompressing the Worker itself isn't an insignificant cost and I assume they preferred decompression speed over size.

I

Isaac McFadyen•2/2/24, 1:17 PM

It would also be a fairly major task to switch over to a new algorithm considering they'd need to decompress and recompress thousands of old Workers and switch all of the runtime components to the new algorithm across thousands of servers, all without any downtime.

Ppeshakoo Hey Dave! if you don't mind could you show me how you did this?? I'm doing somet...

D

dave•2/2/24, 2:12 PM

Hmm lemme check if we actually ended up doing it or not. ;p

Original message was deleted

D

dave•2/2/24, 2:14 PM

Likely they use Workers for some of the Workers supporting infrastructure. Workers only support compressing gzip streams. https://developer.mozilla.org/en-US/docs/Web/API/CompressionStream/CompressionStream

Ppeshakoo Hey Dave! if you don't mind could you show me how you did this?? I'm doing somet...

D

dave•2/2/24, 2:19 PM

ah sorry, we ended up not needing to use ListObjects. That said, I think it should be pretty straightforward, would be a cool blog post if you end up doing it

Original message was deleted

D

dave•2/2/24, 2:20 PM

Yes, but maintaining an-out-of-tree patch to V8 and making a non-standard Web API kinda sucks

Original message was deleted

D

dave•2/2/24, 2:25 PM

pretty sure they did a blog post on it

Original message was deleted

D

dave•2/2/24, 2:28 PM

https://blog.cloudflare.com/deploying-firmware-at-cloudflare-scale-how-we-update-thousands-of-servers-in-more-than-285-cities

And I was thinking cloudflare workers would work because of the low latency overall.

Similar Threads

Similar Threads

Similar Threads