Am I correct in thinking that I should establish variables each time in a given step as if I define

Am I correct in thinking that I should establish variables each time in a given step as if I define them outside of a step it may not work correctly?

e.g. is this the correct approach establishing my DB client in each step as opposed to one client variable in the parent scope?

SSam Am I correct in thinking that I should establish variables each time in a given ...

jordanfinners•11/13/24, 12:48 PM

I've defined mine in the run function and used it in the steps

it's working, whether it's best practice I don't know though

jordanfinners•11/13/24, 12:49 PM

Is there a way to alert on failed workflows?

SamOP•11/13/24, 12:56 PM

I thought you could potentially return from a step also, but not sure if that would work as it would be returning a connection not a normal value.

SamOP•11/13/24, 12:56 PM

Maybe someone from CF can clarify, I am using hyperdrive also as the connection string.

SamOP•11/13/24, 12:59 PM

Also it seems there is an undocumented payload limit on params for workflows.

When I try to pass a base64 in params it will fail to start the workflow every time but will not throw an error when I instantiate the workflow.

akhilravidas•11/13/24, 5:19 PM

Gm! Loving workflows in general but feels like i'm missing the right way to track errors. I get the exception message when i go to the flow instance page and wrangler tail but no stack trace so it takes a bit longer to debug. Any way to get the full stack trace?

SSam I thought you could potentially return from a step also, but not sure if that wo...

Isaac McFadyen•11/13/24, 7:02 PM

Values that are returned need to be serializable, so yes, you'll need to recreate the DB conn every step.

Isaac McFadyen•11/13/24, 7:02 PM

Additionally, your Workflow may be paused/move machines/etc between steps so putting a DB conn in the global scope isn't advised either

SamOP•11/13/24, 7:42 PM

Cheers for confirming yeah thought I read or heard words to the latter.

Hyperdrive should hopefully minimise the additional latency and reuse the connection I guess!

SSam Cheers for confirming yeah thought I read or heard words to the latter. Hyperd...

Isaac McFadyen•11/13/24, 8:13 PM

Yup - and realistically a DB connection is only on the order of several hundreds of ms/single-digit seconds at the most so unless your Workflow is very time-sensitive it probably won't matter that much.

surgery18•11/14/24, 7:05 PM

Is it possible To have a workflow that uses a websocket? Example. Let's say a user posts a comment on my site and I want that shot off to a workflow and the web client to listen to know when that workflow is done (via a websocket). The workflow does its steps needed on the post data and then the last step in the flow I want it to shoot off a websocket message saying it is done and then the web client can pick up that message and do what needs to be done after that. Is this something possible? Or does the client just have to keep polling the workflow for its status till it is done? Thanks!

Original message was deleted

surgery18•11/14/24, 7:26 PM

Ah ok. Well, that could be something you can add to workflow to have the ability to use websockets within workflow.

Jani•11/14/24, 7:59 PM

You can make workflow push updates to Durable Object, which has websocket support

Sameer Ali•11/14/24, 10:33 PM

Any plan on adding option in cloudflare:testcloudflare:test to write test cases? or there is any other way to writing integeration test for steps?

pao ramen•11/15/24, 8:58 AM

Hi there! I've been porting some python code I had using inngest, and some other built with queues, and I have a bunch of feedback that might be useful.

For reference, the code that I've migrated to workflows usually does something like this:

fetches a bunch of resources from the DB
(fanout) for each resource it does an http call to an API and then it saves some data to the DB
(fanin) once everything is done, it makes more changes to the DB

I think this is the more straightforward kind of workflow you can imagine, and yet, I've found some shortcomings implementing them on Cloudflare:

The 256 step limit means that I can't build this as a single workflow. I have to add a queue to implement the "fanout". The alternative is to do a
```
Promise.allSettled
```
```
Promise.allSettled
```
, but then you don't get to retry failed workflow creation.
The http call has a rate limit, and unlike queues, there is no way to specify concurrency/batch size. Since I moved the "fanout" to a queue, this is where I implement the rate limit, but it would be great to be able to describe the whole workflow without queues.
Because of moving the fanout to queues, there is no way to do the fanin part since it can't be known when all the queues have been processed.
There is no way to compose workflows. This means that if you have a piece of workflow that is used in several places, you have to abstract away the steps. It would be great if workflows could return values so a workflow could call another workflow like a step.

Cheers,
Pao

pao ramen•11/15/24, 11:55 AM

One last thing... Is there a workaround to the "inserted row already exists in table while trying to run multiple workflows locally"? I can't have more than one workflow locally

Jeff•11/15/24, 5:55 PM

are you using the same workflow id/name?

JJeff are you using the same workflow id/name?

pao ramen•11/15/24, 10:05 PM

No. It's this issue: https://github.com/cloudflare/workers-sdk/issues/7127#issuecomment-2478815654

GitHub

🐛 BUG: Fatal uncaught kj::Exception: kj/table.c++:49: failed: inser...

Which Cloudflare product(s) does this pertain to? Workers Runtime, Wrangler, Other What version(s) of the tool(s) are you using? 3.83.0 What version of Node are you using? No response What operatin...

Timo Reimann•11/15/24, 10:57 PM

I'm not sure if I missed to see the answer to the following question anywhere: is an instantiated workflow guaranteed to run on a dedicated worker?

Asking to understand how the worker-specific limits (e.g., subrequests) possibly apply.

Evan•11/15/24, 11:52 PM

Any pointers to docs that describe how workflow can be used with workers for platforms?

edit: looks like this is not currently supported, but would love to understand what the timeline looks like. workflow + workers for platforms is exactly what I need and I'll unfortunately have to forgo workflow until the two can compose together and continue to work on my own workflow layer.

pao ramen•11/16/24, 10:06 AM

More feedback:

bug: when triggering a workflow from the UI, the json payload gets stringified (so it does not work at all)
it would be great to be able to retry workflows/steps from the UI

Sskip Hi @Cayter 👋. Would you recommend sticking with Cloudflare for Remix develope...

MissS•11/17/24, 8:08 PM

Remote isn’t possible while doing dev locally. I.e, can’t access remote R2 while running wrangler dev

So workers with assets is convenient but because it’s not fully fledged out, it depends on how “serious” your project requirements.

The eta for doing running wrangler remotely is this quarter, but I wouldn’t trust CF and assume it would be available sometime early next year

Ppao ramen Hi there! I've been porting some python code I had using inngest, and some other...

MissS•11/17/24, 8:21 PM

Apparently, they are working on making workflows have a built in rate limiter. I wouldn’t assume this to be release anytime soon

Without specific numbers than giving advice is difficult but you could potentially do this with breaking up your fanout into multiple workflows. It’s also possible to have a “always on” workflow that manages other workflows although this solution might be less elegant than just using a queue.

Ppao ramen Hi there! I've been porting some python code I had using inngest, and some other...

MissS•11/17/24, 8:22 PM

Can’t you start another workflow from an existing workflow?

MMissS Can’t you start another workflow from an existing workflow?

pao ramen•11/17/24, 9:29 PM

Yes, but unlike steps, you don't get the result

kchro3•11/18/24, 2:21 AM

I have a question about workflow IDs. It supports setting the ID in case there is some external ID like a user ID, but it sounds like you cannot reuse IDs. Doesn't that defeat the purpose of being able to reference a workflow?

Is the recommendation to append a timestamp with the external ID or something for a recurring workflow per user ID?

MMissS Remote isn’t possible while doing dev locally. I.e, can’t access remote R2 while...

skip•11/18/24, 2:58 AM

Thank MissS :). What's the best way to stay up to date on the progress on Cloudflares side?

Ppao ramen One last thing... Is there a workaround to the "inserted row already exists in t...

pao ramen•11/18/24, 8:57 AM

I will unify all workflows in a single one with a switch case, like queues.

SSameer Ali Any plan on adding option in `cloudflare:test` to write test cases? or there is ...

Diogo Ferreira•11/18/24, 9:55 AM

It is on the Roadmap

SamOP•11/18/24, 11:21 AM

Exporting multiple workflows from wrangler.toml seems to cause errors with miniflare locally (and works fine on remote/deployed)

This is similar syntax that I use for queues, but just to double check is this the correct way to add multiple workflows?

[[workflows]]
name = "process-unstructured-nutrition-data"
binding = "PROCESS_UNSTRUCTURED_NUTRITION_DATA"
class_name = "ProcessUnstructuredNutritionData"

[[workflows]]
name = "photo-meal-analysis"
binding = "PHOTO_MEAL_ANALYSIS"
class_name = "PhotoMealAnalysis"

[[workflows]]
name = "text-meal-analysis"
binding = "TEXT_MEAL_ANALYSIS"
class_name = "TextMealAnalysis"

[[workflows]]
name = "process-unstructured-nutrition-data"
binding = "PROCESS_UNSTRUCTURED_NUTRITION_DATA"
class_name = "ProcessUnstructuredNutritionData"

[[workflows]]
name = "photo-meal-analysis"
binding = "PHOTO_MEAL_ANALYSIS"
class_name = "PhotoMealAnalysis"

[[workflows]]
name = "text-meal-analysis"
binding = "TEXT_MEAL_ANALYSIS"
class_name = "TextMealAnalysis"

I can send you the debug logs, but nothing useful apart from some stuff about LLVM symbolizer. Have validated the issue only occurs when I add the additional workflows, does not occur with just one workflow.

edit: I assume its the same as the issue above https://github.com/cloudflare/workers-sdk/issues/7127 let me know if you need anything from me.

GitHub

🐛 BUG: Fatal uncaught kj::Exception: kj/table.c++:49: failed: inser...

makrdev•11/18/24, 12:42 PM

Guys is there any public information on the roadmap? I mean like what to expect when. (I know it’s still in early but I’m very excited that’s why I wanted to learn more)

As far as I understand from the community these are the most expected features or common issues etc.

1) Concurrency settings per workflow or step (maybe other flow control features like throttling, rate limiting etc.)

2) Event based flows like waitUntil API

3) Returning value from workflow and accessing them inside another workflows

4) Local development issues(multiple workflows, id, observability etc)

5) Workflow level clean-up functions such as ‘onFailure’

6) Prioritization on workflow based on key or something similar

7) Limit increase (concurrency, step count etc.)

DaniFoldi•11/18/24, 12:48 PM

Unfortunately I don't think you'll get answers to when type questions, besides ":soontm:"

DaniFoldi•11/18/24, 12:48 PM

but trust me you're not alone, I'm also excited about Workflows and eagerly waiting for some of these

DDaniFoldi but trust me you're not alone, I'm also excited about Workflows and eagerly wait...

makrdev•11/18/24, 12:58 PM

Yeah I know there are a lot of people like us and setting expectations with clear communication would be great in the early stage. Let’s hope that we get answers at least some of them

DaniFoldi•11/18/24, 1:00 PM

I don't disagree, just saying what I think will be realistic
priorities change all the time, and even if they were to commit to a date/timeline, it's not a 100% guarantee, as a few examples have shown

DDaniFoldi I don't disagree, just saying what I think will be realistic priorities change a...

makrdev•11/18/24, 1:14 PM

Yes, you are right but I think nobody expect commitment on a date or something like 100% guarantee on these features/issues. People want to understand the direction and the level of commitment to the product itself. These are not huge things when you have a clear communication. (Not blaming anybody on the team, just saying things can be improved

)

Mmakrdev Yes, you are right but I think nobody expect commitment on a date or something l...

Matt Silverlock•11/18/24, 5:47 PM

People do interpret dates as commitments

MMatt Silverlock People _do_ interpret dates as commitments 🙂

makrdev•11/18/24, 6:08 PM

Hey Matt, let’s forget about the date then.

It would be great if you could inform us on these topics in order to clarify the direction you guys want to go.

Should we expect these features or not? (or any other feature you want to add) Instead of a date, maybe you can share the timeline relative to other features. (like “we are prioritizing this one over this one” etc.)

Mmakrdev Hey Matt, let’s forget about the date then. It would be great if you could inf...

Matt Silverlock•11/18/24, 6:15 PM

I think we've laid these out in this chat - but prioritization does change. We're super early in the beta (only a month in!) and fixing the prioritization means we'd be ignoring folks / the ability to solve problems as they come up.

Mmakrdev Hey Matt, let’s forget about the date then. It would be great if you could inf...

Matt Silverlock•11/18/24, 6:16 PM

Most of these (bar sharing state across Workflows) will likely land in the next 3-4 months. There's not a strict ordering - some of these are easier than others, others need design time/discussion, etc.

I know that's not the answer you want, but it's the answer I'm giving.

MMatt Silverlock Most of these (bar sharing state across Workflows) will likely land in the next ...

makrdev•11/18/24, 6:38 PM

Not quite, but it’s close enough to the answer I was looking for. I appreciate you taking the time to address it. Now, anyone searching for this can find it too.

martinhoest•11/18/24, 11:36 PM

I'm unsure of the best way to implement a fan-out pattern using Workflows. I fetch a list of files, and want to process each one. Should I loop over the files and call step.do, awaiting them all with await Promise.all? Or should I trigger a new "file-process" workflow instance for each of them?

1984 Ford Laser•11/19/24, 2:14 AM

I've only just started looking at Workflows, I was wondering if it would be suitable for a process where one step requires collecting data from dozens of endpoints (DOs in this case) and waiting until all are returned before going to the next step. Would this be possible?

Mmartinhoest I'm unsure of the best way to implement a fan-out pattern using Workflows. I fet...

makrdev•11/19/24, 7:14 AM

You can do both actually and both have different trade-offs. There is a hard limit (256) for maximum number of steps in a single run. So if the list you want to process is bigger than it you can’t. There is no hard limit for workflows (you can run 100 workflows concurrently tho) however you won’t be able to share state between them without external storage (kv etc.)

11984 Ford Laser I've only just started looking at Workflows, I was wondering if it would be suit...

makrdev•11/19/24, 7:18 AM

Yes, you can do. It’s recommended to make a single fetch call in a step in order not to lose its retry logic and persistency.

Mmakrdev You can do both actually and both have different trade-offs. There is a hard lim...

martinhoest•11/19/24, 7:26 AM

thanks, didn't immediately see this hard limit in the docs - but it is there, I just missed it. I have a few thousand, so need to trigger a workflow for each.

Mmartinhoest I'm unsure of the best way to implement a fan-out pattern using Workflows. I fet...

pao ramen•11/19/24, 10:35 AM

unluckily, workflows have limited amount of steps. So only do step.do for each file if you are 100% sure you are not going to have more than 256 files. Regarding triggering a new process for each, it can be done but it does not support batching nor concurrency. What I do to workaround this is one step, and inside I chunk and send batches to a queue. That queue then triggers the workflows.

Also, since having multiple workflows locally does not work, I recommend having a single workflow that dispatches work.

pao ramen•11/19/24, 10:40 AM

This is how I connected the pieces myself. Basically a queue endpoint that will forward events to a workflow. The workflow then dispatches with a given id. It's based on convention (each workflow has an id, and the queue name is that id + '-queue') and it allows me to have some type safety. I then pass an object around, so I can compose workflows as you expect.

[[workflows]] name = "process-unstructured-nutrition-data" binding = "PROCESS_UNSTRUCTURED_NUTRITION_DATA" class_name = "ProcessUnstructuredNutritionData" [[workflows]] name = "photo-meal-analysis" binding = "PHOTO_MEAL_ANALYSIS" class_name = "PhotoMealAnalysis" [[workflows]] name = "text-meal-analysis" binding = "TEXT_MEAL_ANALYSIS" class_name = "TextMealAnalysis"

Am I correct in thinking that I should establish variables each time in a given step as if I define

Similar Threads

Similar Threads

Similar Threads