I have multiple "children workers" that use HTMLRewriter() to change the HTML of response. What I am

I have multiple "children workers" that use HTMLRewriter() to change the HTML of response.
What I am trying to achieve is to trigger multiple workers with "parent worker".
I know you can use service bindings to bind another worker to it, but the problem I have is I need to run multiple workers and modify the responses accordingly.

For example: if I bind two child workers, one will change meta title (worker 1) and one meta description (worker 2).
What I need to achieve is firstly trigger worker1 with parent request, and then trigger worker 2 on response i get from worker 1, so i get all the HTML changes.

I have no problem getting response from worker 1, but I dont know how to combine worker 1 and worker 2 changes. What i tried is passing body of response from worker 1 to worker 2 fetch, like this:

    // Fetch the resource using childWorker1
    const childWorker1Response = await env.childWorker1.fetch(request.clone());

    // Fully consume the response body of childWorker1
    const childWorker1ResponseBody = await childWorker1Response.text();

    // Fetch a resource using childWorker2, passing the response body of childWorker1 as the POST body
    const childWorker2Response = await env.childWorker2.fetch(request.clone(), {
      method: 'POST',
      body: childWorker1ResponseBody
    });

    // Fetch the resource using childWorker1
    const childWorker1Response = await env.childWorker1.fetch(request.clone());

    // Fully consume the response body of childWorker1
    const childWorker1ResponseBody = await childWorker1Response.text();

    // Fetch a resource using childWorker2, passing the response body of childWorker1 as the POST body
    const childWorker2Response = await env.childWorker2.fetch(request.clone(), {
      method: 'POST',
      body: childWorker1ResponseBody
    });

    // Fetch the resource using childWorker1
    const childWorker1Response = await env.childWorker1.fetch(request.clone());

    // Fully consume the response body of childWorker1
    const childWorker1ResponseBody = await childWorker1Response.text();

    // Fetch a resource using childWorker2, passing the response body of childWorker1 as the POST body
    const childWorker2Response = await env.childWorker2.fetch(request.clone(), {
      method: 'POST',
      body: childWorker1ResponseBody
    });

    // Fetch the resource using childWorker1
    const childWorker1Response = await env.childWorker1.fetch(request.clone());

    // Fully consume the response body of childWorker1
    const childWorker1ResponseBody = await childWorker1Response.text();

    // Fetch a resource using childWorker2, passing the response body of childWorker1 as the POST body
    const childWorker2Response = await env.childWorker2.fetch(request.clone(), {
      method: 'POST',
      body: childWorker1ResponseBody
    });

But this does not seem to work.

Do you have any ideas what is wrong?
Any help would be greatly appreciated!

Vvakno I have multiple "children workers" that use HTMLRewriter() to change the HTML of...

Hello, I’m Allie!•3/4/24, 3:03 PM

Is there any reason why this couldn’t be a single Worker?

HHello, I’m Allie!Is there any reason why this couldn’t be a single Worker?

vaknoOP•3/4/24, 3:05 PM

yes because i will need multiple workers to run with this parent workers on different urls. There would be too many combinations to handle

vaknoOP•3/4/24, 3:13 PM

if i have 5 different child workers there will be 120 permutations

vaknoOP•3/4/24, 3:13 PM

this is way too much to track and keep it organised

Vvakno yes because i will need multiple workers to run with this parent workers on diff...

Hello, I’m Allie!•3/4/24, 3:16 PM

I’m not sure I understand. Can you give me an example?

Vvakno I have multiple "children workers" that use HTMLRewriter() to change the HTML of...

Jadugar_Jaggu•3/4/24, 3:18 PM

i think instead of using request.clone() Just use request directly

Jadugar_Jaggu•3/4/24, 3:19 PM

so it will work as many times as you want

JJadugar_Jaggu i think instead of using request.clone() Just use request directly

Hello, I’m Allie!•3/4/24, 3:19 PM

That still wouldn’t combine the changes though

Hello, I’m Allie!•3/4/24, 3:19 PM

Only make two separate copies, and modifying both slightly

HHello, I’m Allie!I’m not sure I understand. Can you give me an example?

vaknoOP•3/4/24, 3:21 PM

ok so i have 100 urls, on some i need to change meta titles, on some i need to change meta description, on some i need to change both, on some i need to change h1 to h2 and so on. I have like 6 seperate html changes i need to change on different urls.

If i want to make workers that cover all combinations i would need to create 720 different workers.

Vvakno ok so i have 100 urls, on some i need to change meta titles, on some i need to c...

Hello, I’m Allie!•3/4/24, 3:22 PM

Exactly, so why not a single Worker that does everything?

HHello, I’m Allie!Exactly, so why not a single Worker that does everything?

Jadugar_Jaggu•3/4/24, 3:23 PM

how you will use htmlrewriter to parse html and store it in a var?

I think we can only store a htmlrewriter response if we use it in a service worker and make a hit from parent

Jadugar_Jaggu•3/4/24, 3:24 PM

is there another way of doing it?

JJadugar_Jaggu how you will use htmlrewriter to parse html and store it in a var? I think we ...

Hello, I’m Allie!•3/4/24, 3:25 PM

Why would you need to store it? Why not just stream it back to the client once it has been modified?

HHello, I’m Allie!Why would you need to store it? Why not just stream it back to the client once i...

Jadugar_Jaggu•3/4/24, 3:26 PM

it depends on the use cases. Maybe he doesn't want to stream it to the client.

Cloudflare workers are very powerful, can do many things with it

HHello, I’m Allie!Exactly, so why not a single Worker that does everything?

vaknoOP•3/4/24, 3:36 PM

because the implementation would look much cleaner and organized code wise, also i think there might be problems with some responses

Vvakno because the implementation would look much cleaner and organized code wise, also...

Hello, I’m Allie!•3/4/24, 3:42 PM

I would note too that this will cost more, since you would have to invoke 2+ Workers

HHello, I’m Allie!I would note too that this will cost more, since you would have to invoke 2+ Wor...

vaknoOP•3/4/24, 3:46 PM

you are right. So basically the best solution is to combine all into one?

Vvakno you are right. So basically the best solution is to combine all into one?

Hello, I’m Allie!•3/4/24, 3:51 PM

Imo, yes. It should also make it (marginally) faster

Rahul Raikwar•3/4/24, 4:06 PM

How can fix this - Error: Failed to publish your Function. Got error: Your Functions script is over the 1 MiB size limit (workers.api.error.script_too_large)

Vvakno ok so i have 100 urls, on some i need to change meta titles, on some i need to c...

Isaac McFadyen•3/4/24, 4:07 PM

There's also a limit of 100 workers on an account so you actually couldn't do this without Workers for Platforms (which generally requires an Enterprise plan).

Isaac McFadyen•3/4/24, 4:07 PM

And I don't think you could bind 720 workers without WfP even if you could create them all.

RRahul Raikwar How can fix this - Error: Failed to publish your Function. Got error: Your Funct...

Chaika•3/4/24, 4:11 PM

Either make your script smaller, or upgrade to Workers Paid which is 10 MiB max

Chaika•3/4/24, 4:11 PM

You're using nextjs or some framework?

Mitya•3/4/24, 4:15 PM

Hey all. If I have a shared JS file, let's say

global.js

global.js

, containing a

{}

{}

, which multiple scripts import and hook things onto, will this leak/bleed into multiple requests to my worker given Node's import caching, or will it be isolated to just the current request?

MMitya Hey all. If I have a shared JS file, let's say `global.js`, containing a `{}`, w...

Isaac McFadyen•3/4/24, 4:22 PM

When Workers are deployed the files are bundled automatically into each Worker, they're not actually imported at runtime. So to answer your question, they'll be completely isolated for each Worker.

However they won't be isolated by each request - if you mutate a variable then that variable will stay mutated on the next request if the user hits the same server, which is likely if the user is the same but is unlikely if two different users or even different browsers are requesting the Worker.

CChaika You're using nextjs or some framework?

Rahul Raikwar•3/4/24, 4:25 PM

Yes next js

IIsaac McFadyen When Workers are deployed the files are bundled automatically into each Worker, ...

Mitya•3/4/24, 4:26 PM

Thanks for the advice, much appreciated. If the leakage occurs within multiple requests for the same user, that's fine, but you're saying I can rely on non-leakage between different users?

Basically my use case is that a request sends a QS param, let's say

foo=1

foo=1

, and I want to hook that onto a global object so other files can use it, rather than laboriously having to pass it around to all functions. But the next request may send QS

foo=2

foo=2

. Sounds like I'll be OK based on what you say. (But presumably in a non-serverless/Worker environment this would be problematic?)

MMitya Thanks for the advice, much appreciated. If the leakage occurs within multiple r...

Hello, I’m Allie!•3/4/24, 4:29 PM

You can’t be guaranteed that it will/won’t ever leak

MMitya Thanks for the advice, much appreciated. If the leakage occurs within multiple r...

Isaac McFadyen•3/4/24, 4:29 PM

So basically a global variable is stored per server (called a "metal"). Cloudflare has many many metals so it's unlikely that one user will hit the same metal, but it's very much possible especially in smaller datacenters with only a few metals. I'd not rely on users hitting different metals and just write any code with the idea that it might leak between users if they hit the same metal.

Chaika•3/4/24, 4:31 PM

(the other way I'd view it is eventually if you scale up/become big enough it's going to happen more and more, unlikely at small scale sure but still bad idea)

Mitya•3/4/24, 4:32 PM

Interesting - thanks very much. So if I can't hook things onto a global object, for the reasons you guys have said, is my only remaining option to explicitly pass around info between concerns (middleware/controllers/helpers) or is there some other way I'm missing?

Mitya•3/4/24, 4:32 PM

(I come from PHP land, where everything is isolated per request, even globals.)

MMitya Interesting - thanks very much. So if I can't hook things onto a global object, ...

Hello, I’m Allie!•3/4/24, 4:35 PM

If it is only per-request storage you need, you could use https://developers.cloudflare.com/workers/runtime-apis/nodejs/asynclocalstorage

Cloudflare Docs

AsyncLocalStorage · Cloudflare Workers docs

Cloudflare Workers provides an implementation of a subset of the Node.js AsyncLocalStorage API for creating in-memory stores that remain coherent …

Mitya•3/4/24, 4:45 PM

Oh, interesting. Presumably good old, synchronous, browser-style

globalObject.localStorage

globalObject.localStorage

wouldn't work? If so, using AsyncLocalStorage, am I right that it would look something like this, to store a global property for later reading downstream?
Upstream file:

import { AsyncLocalStorage } from 'node:async_hooks';
const fooStore = new AsyncLocalStorage();
fooStore.run('bar'); //<-- running in some async middleware

import { AsyncLocalStorage } from 'node:async_hooks';
const fooStore = new AsyncLocalStorage();
fooStore.run('bar'); //<-- running in some async middleware

Downstream file:

import { AsyncLocalStorage } from 'node:async_hooks';
const fooStore = new AsyncLocalStorage();
const fooVal = fooStore.getStore(); //"bar"

import { AsyncLocalStorage } from 'node:async_hooks';
const fooStore = new AsyncLocalStorage();
const fooVal = fooStore.getStore(); //"bar"

JJadugar_Jaggu hey guys is anyone familiar with this issue? I am facing it right after automati...

Chaika•3/4/24, 4:46 PM

Thanks for reporting. I would watch this incident for further updates on the issue, and try again when it is resolved:
https://www.cloudflarestatus.com/incidents/3q7sg3tpp7qv

Issues changing CPU Limit for Cloudflare Workers and Pages

MMitya Oh, interesting. Presumably good old, synchronous, browser-style `globalObject.l...

kian•3/4/24, 4:46 PM

import { AsyncLocalStorage } from "node:async_hooks";

const asyncLocalStorage = new AsyncLocalStorage();

async function doStuff() {
  const { req, env, ctx } = asyncLocalStorage.getStore();

  const obj = await env.KV.get("foo");

  // ...
}

export default {
  async fetch(req, env, ctx) {
    return asyncLocalStorage.run({ req, env, ctx }, async () => {
      await doStuff();

      return new Response("Hello World!");
    });
  },
};

import { AsyncLocalStorage } from "node:async_hooks";

const asyncLocalStorage = new AsyncLocalStorage();

async function doStuff() {
  const { req, env, ctx } = asyncLocalStorage.getStore();

  const obj = await env.KV.get("foo");

  // ...
}

export default {
  async fetch(req, env, ctx) {
    return asyncLocalStorage.run({ req, env, ctx }, async () => {
      await doStuff();

      return new Response("Hello World!");
    });
  },
};

kian•3/4/24, 4:47 PM

run

run

takes the object you want to store, and then a callback

Mitya•3/4/24, 4:49 PM

Thanks @kian . So I was nearly there but just need to pass an object? And presumably the callback is optional if all I'm doing is writing to the store for future use, rather than, as per the docs example, sending a response once write is complete?

kian•3/4/24, 4:49 PM

Doesn't necessarily have to be an object

kian•3/4/24, 4:49 PM

Can be a string, number, array, etc

Mitya•3/4/24, 4:50 PM

Yeah... so my original effort would work? (I passed just a string value there, not an object.) But is the general flow correct? Appreciate the help.

kian•3/4/24, 4:50 PM

run

run

requires the callback

kian•3/4/24, 4:50 PM

The idea is that it runs that function async within a 'context'

kian•3/4/24, 4:50 PM

and the store for that context is what you passed as the first arg

Mitya•3/4/24, 4:51 PM

Ah... then I'm struggling a little. You're saying

mystore.run({foo: bar})

mystore.run({foo: bar})

is insufficient to write to the storage? If so, what would any callback I passed actually "do", if all I want to do is write for later reading?

kian•3/4/24, 4:58 PM

The storage is only available within the function passed to

run

run

kian•3/4/24, 4:59 PM

i.e in my case, I put everything inside the function I pass to

run

run

and return my Response there

kian•3/4/24, 4:59 PM

Everything else can now do

asyncLocalStorage.getStore();

asyncLocalStorage.getStore();

to access req, env, ctx

Mitya•3/4/24, 5:00 PM

OK, that makes sense. But does this mean that downstream files/functions in other files will not be able to access the storage value because they're not running in

run()

run()

's callback?

Say file A (middleware) wants to write a value to storage. That middleware then hands over to a controller (file B), which then calls a helper function in file C. Is it possible files B/C to access the storage value written by file A?

I have multiple "children workers" that use HTMLRewriter() to change the HTML of response. What I am

Similar Threads

Similar Threads

Similar Threads