I have a growing backlog of pending workflow instances, while there are only a dozen (not 100) curre

RRakhim I have a growing backlog of pending workflow instances, while there are only a d...

M

Matt Silverlock•1/16/25, 12:59 PM

Can you share your account ID here? (it’s safe to share publicly)

MMatt Silverlock Can you share your account ID here? (it’s safe to share publicly)

R

RakhimOP•1/16/25, 1:17 PM

1ac93aea0842f2d3ef10111b5bf7bb11

1ac93aea0842f2d3ef10111b5bf7bb11

K

kchro3•1/16/25, 11:02 PM

Hi team, can you clarify if "concurrent executions" means any incomplete workflow or if it means any concurrently running steps?

For example, if I have a workflow with delays of multiple days, would they be considered concurrent while they are sleeping?

Kkchro3 Hi team, can you clarify if "concurrent executions" means any incomplete workflo...

T

tadhglewis•1/17/25, 10:25 AM

No, it's only running instances. Sleeping ones aren't counted

M

Mr. Technician•1/17/25, 3:44 PM

can I call workflow steps in a loop?

MMr. Technician can I call workflow steps in a loop?

A

Ashley•1/17/25, 6:43 PM

Yup! It’s just vanilla JS/TS, so you can do loops/conditionals etc

AAshley Yup! It’s just vanilla JS/TS, so you can do loops/conditionals etc

M

Mr. Technician•1/17/25, 6:44 PM

great, I wanted to make sure that wouldn't upset anything in the step architecture

M

Mr. Technician•1/17/25, 6:45 PM

the only thing cloudflare workers is missing is support for puppeteer stealth plugins when using the browser rendering API

M

Mr. Technician•1/18/25, 5:09 PM

is this a reasonable way of making sure this variable remains accessible?

const maxFutureDays = await step.do(`get ${key} max days`, async () => 330);

const maxFutureDays = await step.do(`get ${key} max days`, async () => 330);

const maxFutureDays = await step.do(`get ${key} max days`, async () => 330);

const maxFutureDays = await step.do(`get ${key} max days`, async () => 330);

AAshley Yup! It’s just vanilla JS/TS, so you can do loops/conditionals etc

M

Mr. Technician•1/18/25, 5:12 PM

what I'm not clear on here is what happens if the engine restarts while my loop is running

M

Mr. Technician•1/18/25, 5:13 PM

for example I have a while loop that I will break out of depending on the result of the step called within it

M

Mr. Technician•1/18/25, 5:13 PM

I guess I am safe as long as all of my variables are step returns

M

Mr. Technician•1/18/25, 5:18 PM

also, I'm not sure my question was clear. I want to call steps within a loop, not use loops within a step

M

Mr. Technician•1/18/25, 5:18 PM

for example:

M

Mr. Technician•1/18/25, 5:19 PM

while (somecondition){
    await step.do(`some action `${somekey}`, async () => {});
}

while (somecondition){
    await step.do(`some action `${somekey}`, async () => {});
}

while (somecondition){
    await step.do(`some action `${somekey}`, async () => {});
}

while (somecondition){
    await step.do(`some action `${somekey}`, async () => {});
}

M

Mr. Technician•1/18/25, 5:19 PM

but this seems to violate: https://developers.cloudflare.com/workflows/build/rules-of-workflows/#name-steps-deterministically

Cloudflare Docs

Rules of Workflows · Cloudflare Workflows docs

A Workflow contains one or more steps. Each step is a self-contained, individually retriable component of a Workflow. Steps may emit (optional) state that allows a Workflow to persist and continue from that step, even if a Workflow fails due to a network or infrastructure issue.

M

Mr. Technician•1/18/25, 5:20 PM

the problem is my workflow isn't strictly deterministic, I need to check state in D1, then depending on the result from my upstream API I need to loop another step an unknown number of times

M

Mr. Technician•1/18/25, 5:21 PM

the documentation seems to discourage this

A

Ashley•1/18/25, 5:22 PM

It’s hard without knowing the underlying data, but the step name as you linked is the cache key. So as long as the data points you’re using are determinsticly named, it will just return from cache if that step name has a value. If you’re looping over results in D1, why not use the table’s primary key for example in the step name?

AAshley It’s hard without knowing the underlying data, but the step name as you linked i...

M

Mr. Technician•1/18/25, 5:27 PM

I can provide a little more context. I'm the developer of https://railforless.us/ which scrapes prices for Amtrak's trains. They use a bucket pricing system and I am working on a workflow that will find every current bucket (I know there are 8). I am scraping 14 days of prices per step, so I would loop until I have found all 8 buckets (or give up after x tries). It sounds like I could use the date range as the key and it would correctly cache the state of that step?

RailForLess.us

Taking the train shouldn't be a pain. Make your next Amtrak trip a reality with flexible scheduling and the tools you need to plan the perfect trip.

A

Ashley•1/18/25, 5:39 PM

Yes, I think so, as long as the date is consistent (e.g. not using “now” functions where the time changes etc so make sure it’s just YYYY-MM-DD or something)

M

Mr. Technician•1/18/25, 11:29 PM

it would be, yeah

MMatt Silverlock Can you share your account ID here? (it’s safe to share publicly)

R

RakhimOP•1/19/25, 10:28 AM

any updates? I had to reduce the pace of creating new workflow instances in an attempt to reduce the backlog, it works but very slowly. I'm still never seeing anything close to 100 concurrent workflows

MMr. Technician I can provide a little more context. I'm the developer of https://railforless.us...

T

thousandmiles•1/20/25, 5:39 AM

great tool you built. so is your website purely serverless on cf right now?

Tthousandmiles great tool you built. so is your website purely serverless on cf right now?

M

Mr. Technician•1/20/25, 5:41 AM

thanks! My buddy takes credit for the interface while i handled all of the API pieces. But yes, it’s purely serverless on cf with the exception of our remote puppeteer service, since the browser rendering API in workers is missing features we need

M

Mr. Technician•1/20/25, 5:43 AM

i use websockets, durable objects, queues, d1, and cron jobs

M

Mr. Technician•1/20/25, 5:43 AM

and kv

MMr. Technician thanks! My buddy takes credit for the interface while i handled all of the API p...

T

thousandmiles•1/20/25, 5:45 AM

impressive! though i am not quite proficient on cf worker myself, but i did manage some heavy workload using aws lambda/step functions back in 2 years.

M

Mr. Technician•1/20/25, 5:46 AM

i originally wrote the backend in .net to run in azure functions, then rewrote it all in typescript to leverage workers

MMr. Technician i use websockets, durable objects, queues, d1, and cron jobs

T

thousandmiles•1/20/25, 5:47 AM

are you planning to opensource (even parts of) this backend as well?

MMr. Technician i originally wrote the backend in .net to run in azure functions, then rewrote i...

T

thousandmiles•1/20/25, 5:48 AM

way to go. this is exactly i am doing now, u know, think serverless. even LLM/agentic workflow is going serverless

M

Mr. Technician•1/20/25, 5:48 AM

possibly. the react front-end is open-source presently

M

Mr. Technician•1/20/25, 5:49 AM

i hope my “secret” is kept safe from amtrak who would probably rather this service didn’t exist

M

Mr. Technician•1/20/25, 5:50 AM

it’s quite frankly only possible because of how horribly designed their booking system is

J

Jikyu•1/20/25, 6:06 AM

Can a workflow call itself? Like how a worker can call itself via service binding

MMr. Technician i hope my “secret” is kept safe from amtrak who would probably rather this servi...

T

thousandmiles•1/20/25, 6:08 AM

haha, that makes sense.

L

Luca (steebchen)•1/20/25, 9:50 AM

it would be awesome to have full text search on the params on all workflows as well

E

eduardo | Latitude.sh•1/20/25, 12:42 PM

Error handling seems quite complex. I have a tail worker that should post to the bug tracking app, but it doesn't seem to catch exceptions at all

tail worker excerpt:

  async tail(events, env, ctx) {
    // Check if events array exists and has items
    if (!Array.isArray(events) || events.length === 0) {
      return new Response("No events found", { status: 200 });
    }

    // Only proceed if there's an exception AND exceptions array is not empty
    const event = events[0];
    if (event.outcome !== "exception" || !event.exceptions?.length) {
      return new Response("No exception found", { status: 200 });
    }

    // Log the event structure to help us understand the data
    console.log("Tail worker event:", JSON.stringify(events, null, 2));

...

  async tail(events, env, ctx) {
    // Check if events array exists and has items
    if (!Array.isArray(events) || events.length === 0) {
      return new Response("No events found", { status: 200 });
    }

    // Only proceed if there's an exception AND exceptions array is not empty
    const event = events[0];
    if (event.outcome !== "exception" || !event.exceptions?.length) {
      return new Response("No exception found", { status: 200 });
    }

    // Log the event structure to help us understand the data
    console.log("Tail worker event:", JSON.stringify(events, null, 2));

...

  async tail(events, env, ctx) {
    // Check if events array exists and has items
    if (!Array.isArray(events) || events.length === 0) {
      return new Response("No events found", { status: 200 });
    }

    // Only proceed if there's an exception AND exceptions array is not empty
    const event = events[0];
    if (event.outcome !== "exception" || !event.exceptions?.length) {
      return new Response("No exception found", { status: 200 });
    }

    // Log the event structure to help us understand the data
    console.log("Tail worker event:", JSON.stringify(events, null, 2));

...

  async tail(events, env, ctx) {
    // Check if events array exists and has items
    if (!Array.isArray(events) || events.length === 0) {
      return new Response("No events found", { status: 200 });
    }

    // Only proceed if there's an exception AND exceptions array is not empty
    const event = events[0];
    if (event.outcome !== "exception" || !event.exceptions?.length) {
      return new Response("No exception found", { status: 200 });
    }

    // Log the event structure to help us understand the data
    console.log("Tail worker event:", JSON.stringify(events, null, 2));

...

workflow step

        const payload = await step.do('prepares payload', async () => {
            const isDev = this.env.ENVIRONMENT === 'dev';
            const data = isDev ? mockData : event.payload;
            
            console.log("Received payload:", JSON.stringify(data, null, 2));
            throw new Error("Testing error handling with payload data");
            
            return data;
        });

        const payload = await step.do('prepares payload', async () => {
            const isDev = this.env.ENVIRONMENT === 'dev';
            const data = isDev ? mockData : event.payload;
            
            console.log("Received payload:", JSON.stringify(data, null, 2));
            throw new Error("Testing error handling with payload data");
            
            return data;
        });

        const payload = await step.do('prepares payload', async () => {
            const isDev = this.env.ENVIRONMENT === 'dev';
            const data = isDev ? mockData : event.payload;
            
            console.log("Received payload:", JSON.stringify(data, null, 2));
            throw new Error("Testing error handling with payload data");
            
            return data;
        });

        const payload = await step.do('prepares payload', async () => {
            const isDev = this.env.ENVIRONMENT === 'dev';
            const data = isDev ? mockData : event.payload;
            
            console.log("Received payload:", JSON.stringify(data, null, 2));
            throw new Error("Testing error handling with payload data");
            
            return data;
        });

What am I missing? It never throws the error

L

Lila Rest•1/20/25, 3:58 PM

Hey there! Is there any possibility of running some of the steps of a workflow in parallel? For example, in different worker invocations to minimize the overall execution speed?

LLila Rest Hey there! Is there any possibility of running some of the steps of a workflow i...

M

Matt Silverlock•1/21/25, 4:05 PM

You can always fire off another Workflow, even from within a Workflow. Workflows are the way to run tasks concurrently, or in parallel.

B

Bonadio•1/21/25, 6:36 PM

Hi, what is the use of the return of the run function in the workflow?
I saw in the twillio example that it returns a value.
https://github.com/craigsdennis/twilio-cloudflare-workflow/blob/main/src/index.ts

export class NeverGonnaWorkflow extends WorkflowEntrypoint<Env, NeverGonnaParams> {
    async run(event: WorkflowEvent<NeverGonnaParams>, step: WorkflowStep) {
        const { to, host, content } = event.payload;
        await step.sleep('wait for the right moment', '135 seconds');
        const callSid = await step.do('call person back', async () => {
            const client = new Twilio(this.env.TWILIO_ACCOUNT_SID, this.env.TWILIO_AUTH_TOKEN);
            // Say verb here does text to speech
            // Play allows you to play media.
            // The mp3 file is hosted on Cloudflare Workers using dynamic assets
            const twiml = `
            <Response>
                <Say>Hello from a Cloudflare Workflow!</Say>
                <Say>You said "${content}".</Say>
                <Say>Check out this classic:</Say>
                <Play>https://${host}/classic.mp3</Play>
            </Response>`;
            const call = await client.calls.create({
                to,
                from: this.env.TWILIO_PHONE_NUMBER,
                twiml,
            });
            return call.sid;
        });

        return { success: true, callSid };   // <------ How to use this?

    }
}

export class NeverGonnaWorkflow extends WorkflowEntrypoint<Env, NeverGonnaParams> {
    async run(event: WorkflowEvent<NeverGonnaParams>, step: WorkflowStep) {
        const { to, host, content } = event.payload;
        await step.sleep('wait for the right moment', '135 seconds');
        const callSid = await step.do('call person back', async () => {
            const client = new Twilio(this.env.TWILIO_ACCOUNT_SID, this.env.TWILIO_AUTH_TOKEN);
            // Say verb here does text to speech
            // Play allows you to play media.
            // The mp3 file is hosted on Cloudflare Workers using dynamic assets
            const twiml = `
            <Response>
                <Say>Hello from a Cloudflare Workflow!</Say>
                <Say>You said "${content}".</Say>
                <Say>Check out this classic:</Say>
                <Play>https://${host}/classic.mp3</Play>
            </Response>`;
            const call = await client.calls.create({
                to,
                from: this.env.TWILIO_PHONE_NUMBER,
                twiml,
            });
            return call.sid;
        });

        return { success: true, callSid };   // <------ How to use this?

    }
}

export class NeverGonnaWorkflow extends WorkflowEntrypoint<Env, NeverGonnaParams> {
    async run(event: WorkflowEvent<NeverGonnaParams>, step: WorkflowStep) {
        const { to, host, content } = event.payload;
        await step.sleep('wait for the right moment', '135 seconds');
        const callSid = await step.do('call person back', async () => {
            const client = new Twilio(this.env.TWILIO_ACCOUNT_SID, this.env.TWILIO_AUTH_TOKEN);
            // Say verb here does text to speech
            // Play allows you to play media.
            // The mp3 file is hosted on Cloudflare Workers using dynamic assets
            const twiml = `
            <Response>
                <Say>Hello from a Cloudflare Workflow!</Say>
                <Say>You said "${content}".</Say>
                <Say>Check out this classic:</Say>
                <Play>https://${host}/classic.mp3</Play>
            </Response>`;
            const call = await client.calls.create({
                to,
                from: this.env.TWILIO_PHONE_NUMBER,
                twiml,
            });
            return call.sid;
        });

        return { success: true, callSid };   // <------ How to use this?

    }
}

export class NeverGonnaWorkflow extends WorkflowEntrypoint<Env, NeverGonnaParams> {
    async run(event: WorkflowEvent<NeverGonnaParams>, step: WorkflowStep) {
        const { to, host, content } = event.payload;
        await step.sleep('wait for the right moment', '135 seconds');
        const callSid = await step.do('call person back', async () => {
            const client = new Twilio(this.env.TWILIO_ACCOUNT_SID, this.env.TWILIO_AUTH_TOKEN);
            // Say verb here does text to speech
            // Play allows you to play media.
            // The mp3 file is hosted on Cloudflare Workers using dynamic assets
            const twiml = `
            <Response>
                <Say>Hello from a Cloudflare Workflow!</Say>
                <Say>You said "${content}".</Say>
                <Say>Check out this classic:</Say>
                <Play>https://${host}/classic.mp3</Play>
            </Response>`;
            const call = await client.calls.create({
                to,
                from: this.env.TWILIO_PHONE_NUMBER,
                twiml,
            });
            return call.sid;
        });

        return { success: true, callSid };   // <------ How to use this?

    }
}

GitHub

twilio-cloudflare-workflow/src/index.ts at main · craigsdennis/twil...

Use Cloudflare Workers to create a scheduled durable execution with your Twilio applications - craigsdennis/twilio-cloudflare-workflow

BBonadio Hi, what is the use of the return of the run function in the workflow? I saw in...

M

Matt Silverlock•1/21/25, 6:51 PM

You can capture that value when you inspect an instances’ status. This can be useful if your Workflow is computing some kind of result, storing data against a key, or generating some kind of identifier

MMatt Silverlock You can always fire off another Workflow, even from within a Workflow. Workflows...

L

Lila Rest•1/22/25, 8:31 AM

Thanks Matt! And will this new workflow run in the same thread or in another?
I know that workers called from other workers will run on the same resources. We are executing a huge tree of operations and I’m trying to maximize the speed of execution.

LLila Rest Thanks Matt! And will this new workflow run in the same thread or in another? I ...

M

Matt Silverlock•1/22/25, 11:03 AM

That wont affect it here.

MMatt Silverlock That wont affect it here.

L

Lila Rest•1/22/25, 3:31 PM

To be 100% sure you're saying that calling a workflow from another will happen in two separate threads?

LLila Rest To be 100% sure you're saying that calling a workflow from another will happen i...

M

Matt Silverlock•1/22/25, 3:39 PM

I'm saying it's no different from any other Worker invocation. If the Workflow is a different Workflow (script) then it is likely to spawn a new isolate. We don't make any more guarantees than that.

MMark Hey all. Does anyone know what token permission is required to create a new wor...

M

Matt Silverlock•1/23/25, 5:59 PM

Workers scripts edit + read:

MMatt Silverlock I'm saying it's no different from any other Worker invocation. If the Workflow i...

L

Lila Rest•1/24/25, 9:08 AM

Ah I see, all clear!

MMatt Silverlock Can you share your account ID here? (it’s safe to share publicly)

M

m.muiznieks•1/26/25, 3:08 PM

Yes, I've noticed the same thing for me as well.
Also the response that request is queued comes after 60-80s.

The time until START of the step since receiving is 40+ minutes. :))

(thats the reason I came here for looking is this an issue for anyone else).

My account ID: 5e3dea7b807381cba36b9292bde14920

Additionally I can say that even sending the requests to my workflow worker gets even 60 to 90 seconds to receive any response from them. Is this a feature? I was not aware of this before.

I have a growing backlog of pending workflow instances, while there are only a dozen (not 100) curre

Similar Threads

Similar Threads

Similar Threads