Cloudflare Developers•13mo ago

I'll check it out next week 🧑‍🍳

I'll check it out next week

D Trombett•12/13/24, 11:11 PM

One thing that imho could be very cool too is a method to list all the running workflows

DD Trombett One thing that imho could be very cool too is a method to list all the running w...

Matt Silverlock•12/14/24, 1:56 PM

Right now you can also use the REST API (but something we can look into on the binding)

DD Trombett One thing that imho could be very cool too is a method to list all the running w...

Matt Silverlock•12/14/24, 1:56 PM

https://developers.cloudflare.com/api-next/resources/workflows/subresources/instances/

the JS SDK

Cloudflare API | Workflows › Instances

Interact with Cloudflare's products and services via the Cloudflare API

Matt Silverlock•12/14/24, 1:59 PM

import Cloudflare from 'cloudflare';

const client = new Cloudflare({
  apiToken: this.env.CLOUDFLARE_API_TOKEN
});

for await (const instance of client.workflows.instances.list('my-workflow', { account_id: this.env.ACCOUNT_ID, status: "running"})) {
  console.log(instance.id);
}

import Cloudflare from 'cloudflare';

const client = new Cloudflare({
  apiToken: this.env.CLOUDFLARE_API_TOKEN
});

for await (const instance of client.workflows.instances.list('my-workflow', { account_id: this.env.ACCOUNT_ID, status: "running"})) {
  console.log(instance.id);
}

MMatt Silverlock ```ts import Cloudflare from 'cloudflare'; const client = new Cloudflare({ ap...

D Trombett•12/14/24, 3:10 PM

Thanks, this looks perfect!

Mozzy•12/14/24, 6:29 PM

I have a non-Workflow function that fetches, validates with Zod, and then puts data to KV.
I was thinking I could break up each step into a Workflow step, but unfortunately, the data is well above the limit: Serialized RPC arguments or return values are limited to 1MiB, but the size of this value was: 4972045 bytes.Serialized RPC arguments or return values are limited to 1MiB, but the size of this value was: 4972045 bytes.

Do I just need to put this all into 1 step to make it work?

Mozzy•12/14/24, 6:33 PM

But then I also feel like I lose parts of what Workflows are

Mozzy•12/14/24, 7:02 PM

Would it make sense to store the data in an array outside the steps like so:

Declare variable
In Step 1: fetch data and store in variable
In Step 2: use the data from the variable

MMozzy Would it make sense to store the data in an array outside the steps like so: 1. ...

Isaac McFadyen•12/14/24, 7:04 PM

No because state outside a step may be cleared at any time: https://developers.cloudflare.com/workflows/build/rules-of-workflows/#do-not-rely-on-state-outside-of-a-step

MMozzy But then I also feel like I lose parts of what Workflows are

Isaac McFadyen•12/14/24, 7:06 PM

Not really, because that concept is idempotent so can be put in one step:

fetch fails, so both ways would fail anyway
zod validation fails, which causes a failure in both scenarios
insert into KV fails in which case you can safely fetch again

Mozzy•12/14/24, 7:07 PM

It's more that I would like to have retries on the fetch part only

Mozzy•12/14/24, 7:09 PM

Like, if the KV or validation fails, I don't want to redo the fetching
I know I can implement this myself within the step, but I'm looking to learn Workflows and that would feel unintuitive

MMozzy It's more that I would like to have retries on the fetch part only

Isaac McFadyen•12/14/24, 7:12 PM

How expensive is the fetch?

IIsaac McFadyen How expensive is the fetch?

Mozzy•12/14/24, 7:14 PM

It's to a resource I don't own so I'd rather not do any more than I need to. It's also several fetches per run

Carl•12/14/24, 7:20 PM

is there a recommended number of workflows per worker. the indefinite queued state is becoming an issue so i'm exploring have each workflow with it's own worker (not an ideal situation and not sure if it might improve things).

peachneo•12/15/24, 4:55 AM

is the concurrent execution limit likely to be raised significantly (like over 100k) once the beta is complete? (Trying to figure out if I'm better off using DOs or workflows)

yzziK•12/15/24, 3:35 PM

Hey so i have a rag workflow pipeline for a hackathon the issue is the steps are failing with
"Too many subrequests." for generating embeddings and storing it into neon db(its a requirement thats why i cant use service binding to D1). are there any subrequest limits in workflows ??

-free tier account

yzziK•12/15/24, 4:34 PM

Ok found it 50 subrequests only

YyzziK Hey so i have a rag workflow pipeline for a hackathon the issue is the steps are...

Matt Silverlock•12/15/24, 5:03 PM

Same as Workers: 50 on free, 1000 on Paid.

YyzziK Hey so i have a rag workflow pipeline for a hackathon the issue is the steps are...

Matt Silverlock•12/15/24, 5:04 PM

https://developers.cloudflare.com/workflows/reference/limits/

Cloudflare Docs

Limits · Cloudflare Workflows docs

Limits that apply to authoring, deploying, and running Workflows are detailed below.

yzziK•12/15/24, 5:04 PM

Thats so bad

yzziK•12/15/24, 5:04 PM

I think I'll have to invoke other workflows from the current one to bypass it

yzziK•12/15/24, 5:04 PM

I hope ?

YyzziK I think I'll have to invoke other workflows from the current one to bypass it

Matt Silverlock•12/15/24, 5:05 PM

You still have a daily limit on total invocations (it's a free plan). 50 subrequests per Workflow instance is still a fair few.

yzziK•12/15/24, 5:07 PM

It's 100 invocations per day right. I need to process 50~200 documents at max so maybe send them in a batch of 25-40 to other workflows

YyzziK It's 100 invocations per day right. I need to process 50~200 documents at max so...

Matt Silverlock•12/15/24, 5:08 PM

It’s 100,000 per day (see the docs I linked)

yzziK•12/15/24, 5:09 PM

Nice

yzziK•12/15/24, 9:29 PM

Can I call the worker's endpoint from the workflow in the same application? How do I use service binding in the same worker.

Does using a service binding like this, I can avoid reaching 50 subrequests limit ?

peachneo•12/16/24, 3:21 AM

Keen to understand what kind of limit is envisioned longer term for "Concurrent Workflow instances" if anyone from CF can speak to that?

oof2win2•12/16/24, 9:58 AM

i don't mean to be rude in any way, but is there any ETA on the ghost executions? i really find workflows to be the best thing for my usecase, but i can't use them if a certain amount of executions just get killed and not even retried

oof2win2•12/16/24, 10:00 AM

or if there is anything on my side that i can do to resolve the issue i'd be happy to do so

Ppeachneo Keen to understand what kind of limit is envisioned longer term for "Concurrent ...

Matt Silverlock•12/16/24, 1:43 PM

For Paid: ballpark 1k-5k per Workflow.

Ooof2win2 i don't mean to be rude in any way, but is there any ETA on the ghost executions...

Diogo Ferreira•12/16/24, 1:45 PM

We are aware of such phenomenon and we are actively working on a fix. Since last week, the chance of getting a ghost execution should have reduced significantly. Further, we are currently testing mechanisms for the system to evict ghost instances. More updates will arrive soon.

oof2win2•12/16/24, 1:47 PM

Thank you so much!

MMatt Silverlock For Paid: ballpark 1k-5k per Workflow.

peachneo•12/16/24, 11:24 PM

Thanks good to know - Out of interest, when I look at the DO API, I don't immediately see why it would not be unlimited eventually? Curious what the constraints are

Ppeachneo Thanks good to know - Out of interest, when I look at the DO API, I don't immedi...

Matt Silverlock•12/17/24, 12:15 AM

We’re still assessing limits. Workflows isn’t just a single DO - we’re doing all of the orchestration, metrics, etc.

Tushar•12/17/24, 2:45 PM

can i write workflows in rust

TTushar can i write workflows in rust

Matt Silverlock•12/17/24, 4:47 PM

Not yet. Something we want to land closer to the GA. Focusing on the foundations first.

Tushar•12/17/24, 5:01 PM

gotcha, good luck ^^

DeChill•12/17/24, 7:07 PM

Are there any updates on getting the Workflow return value through the binding, or Is that on the roadmap?
Adding the Cloudflare API SDK just to get the workflow return value seems to be a bit too much.

I currently have to do a workaround by adding an additional unnecessary step to the end of my workflows, just so I can access the value from the InstanceStatusInstanceStatus outputoutput property, which only contains an array with the individual step outputs.

DDeChill Are there any updates on getting the Workflow return value through the binding, ...

Matt Silverlock•12/17/24, 7:47 PM

What do you consider the "return" value here? Steps return values. The runrun method does not, so there's not a "singular" value.

MMatt Silverlock What do you consider the "return" value here? Steps return values. The `run` met...

DeChill•12/17/24, 7:55 PM

Thanks for the reply Matt. The workflow run method definition has an return value that you can type (it isn't defined as Promise<void>) and the Workflow output (what the runrun method returns) is also prominently shown on the Workflow dashboard. So I would assume this is an intended feature of workflows.

DeChill•12/17/24, 7:59 PM

In the Cloudflare API "Get Logs And Status From Instance" endpoint, the workflow output is also returned separately from the "steps" results. But confusingly, the outputoutput property of the Binding InstanceStatusInstanceStatus contains the step results, not the workflow output.

DDeChill Thanks for the reply Matt. The workflow run method definition has an return valu...

Matt Silverlock•12/17/24, 8:19 PM

Not quite - something we're iterating on, specifically because returning a single (size limited) string can be confusing here.

MMatt Silverlock Not quite - something we're iterating on, specifically because returning a singl...

DeChill•12/17/24, 8:20 PM

hmm okay, from my testing the type of the API docs are wrong. It does indeed return an object value

DDeChill In the Cloudflare API "Get Logs And Status From Instance" endpoint, the workflow...

SeekerdasbatatasOP•12/17/24, 8:20 PM

But confusingly, the output property of the Binding InstanceStatus contains the step results, not the workflow output.

You mean in local dev, right? what's the behaviour on a real workflow?

SSeekerdasbatatas > But confusingly, the output property of the Binding InstanceStatus contains th...

DeChill•12/17/24, 8:23 PM

hmm it seems like you are right! I just tested it again with my deployed workflow and I do indeed get the workflow output there.

DDeChill hmm okay, from my testing the type of the API docs are wrong. It does indeed ret...

SeekerdasbatatasOP•12/17/24, 8:23 PM

sorry about that

it should be an unknown there - i'll look into it but I think it's an openapi 3.0 limitation

I'll check it out next week 🧑‍🍳

Similar Threads