Exactly, I had an idea to make an abstraction on top of workflows to save every step to R2 and lazil

SSeekerdasbatatas If you mean the workflow instance payload ~1MiB

D

dragonli•10/25/24, 8:36 AM

ok

SSeekerdasbatatas If you mean the workflow instance payload ~1MiB

D

DoggeSlapper•10/25/24, 8:47 AM

is there a simple way to let's say deploy hono into multiple workers?

D

DoggeSlapper•10/25/24, 8:47 AM

rn my build is quite small about 40kb but not sure how big things will get

D

DoggeSlapper•10/25/24, 8:48 AM

but well on paid plan there is bigger limit right?

SSeekerdasbatatas Exactly, I had an idea to make an abstraction on top of workflows to save every ...

A

Advany•10/25/24, 9:05 AM

looping at the code, you would not be able to emake an abstraction right. Because RPC is the only way to save it outside the step... So it would be creating something inside the step to 'save to r2' and 'get from r2'. It wouldn't be easier that just using the binding haha. Or am I missing something?

Ddave Any technical downside to using custom IDs instead of the randomly generated one...

M

Matt Silverlock•10/25/24, 9:31 AM

None.

L

LordSilver•10/25/24, 10:37 AM

Isn't this an oxymoron?

L

LordSilver•10/25/24, 10:37 AM

So you pay for CPU time, but you don't pay for Subrequests.

L

LordSilver•10/25/24, 10:39 AM

So if in my workflow I use fetch to get data from a website and that request takes 3s, I'm billed for that 3s of CPU time, so it's like I paid for the subrequest.

LLordSilver So if in my workflow I use fetch to get data from a website and that request tak...

F

fry69•10/25/24, 10:39 AM

Wall time (e.g. awaiting a fetch request) is not CPU time.

L

LordSilver•10/25/24, 10:40 AM

I guess that's a Cloudflare thing then, because your Worker would still be active while waiting

LLordSilver I guess that's a Cloudflare thing then, because your Worker would still be activ...

F

fry69•10/25/24, 10:41 AM

Yes, but in a sleeping state and not eating billable CPU cycles.

M

MissS•10/25/24, 10:42 AM

Does anyone know how invocation count works with workflows? Does a retry count as a new invocation? Are each step a seperate invocation?

Is a three step workflow that succeeds on the first try, three invocations (because there are three steps), or just a single invocation?

MMissS Does anyone know how invocation count works with workflows? Does a retry count a...

F

fry69•10/25/24, 10:48 AM

From how I read the documentation each workflow is a single invocation and steps do not count as another invocation.

F

fry69•10/25/24, 10:48 AM

But I'd love to hear this confirmed.

F

fry69•10/25/24, 10:49 AM

(This should also be easily testable, just watch how the daily worker count changes on a free account)

M

MissS•10/25/24, 10:50 AM

Oh also, is there a way to limit concurrency of a specific workflow?

Let's say, downstream api is 100 req/min, I'd want to prioritize the oldest workflows and also limit concurrency. How would this look like with workflows?

MMissS Oh also, is there a way to limit concurrency of a specific workflow? Let's say,...

F

fry69•10/25/24, 10:51 AM

I do not think you have control over that part.

F

fry69•10/25/24, 10:51 AM

Unless you do some fancy self-organizing inside the workflow, maybe?

DDmitry I might be able to even swap out Temporal for Cloudflare Workflows 😮

D

Diogo Ferreira•10/25/24, 11:01 AM

Already identified issue @Dmitry . Nevertheless, thank you for reporting!

Ffry69 Unless you do some fancy self-organizing inside the workflow, maybe?

M

MissS•10/25/24, 11:07 AM

You can start a workflow from a queue, and you can limit the consumer concurrency from a queue, so I imagine you can limit the throughput of your workflow like this somehow but the DX probably isn't great

D

DanGamble•10/25/24, 11:36 AM

Are you able to currently export multiple workflows?

This seems to work

[[workflows]]
name = "anya-hindmarch-swatch-import"
binding = "SWATCH_IMPORT"
class_name = "SwatchImport"

[[workflows]]
name = "anya-hindmarch-swatch-import"
binding = "SWATCH_IMPORT"
class_name = "SwatchImport"

[[workflows]]
name = "anya-hindmarch-swatch-import"
binding = "SWATCH_IMPORT"
class_name = "SwatchImport"

[[workflows]]
name = "anya-hindmarch-swatch-import"
binding = "SWATCH_IMPORT"
class_name = "SwatchImport"

But this doesn't:

[[workflows]]
name = "anya-hindmarch-swatch-import"
binding = "SWATCH_IMPORT"
class_name = "SwatchImport"

[[workflows]]
name = "anya-hindmarch-foil-import"
binding = "FOIL_IMPORT"
class_name = "FoilImport"

[[workflows]]
name = "anya-hindmarch-swatch-import"
binding = "SWATCH_IMPORT"
class_name = "SwatchImport"

[[workflows]]
name = "anya-hindmarch-foil-import"
binding = "FOIL_IMPORT"
class_name = "FoilImport"

[[workflows]]
name = "anya-hindmarch-swatch-import"
binding = "SWATCH_IMPORT"
class_name = "SwatchImport"

[[workflows]]
name = "anya-hindmarch-foil-import"
binding = "FOIL_IMPORT"
class_name = "FoilImport"

[[workflows]]
name = "anya-hindmarch-swatch-import"
binding = "SWATCH_IMPORT"
class_name = "SwatchImport"

[[workflows]]
name = "anya-hindmarch-foil-import"
binding = "FOIL_IMPORT"
class_name = "FoilImport"

I proceed to get the attached error and then the

.log

.log

gives me no help outside of saying it requires a

.env

.env

. I have

.dev.vars

.dev.vars

. If I copy these to

.env

.env

as well I get no additional errors at all in the log but it still crashes with the same message as the first screenshot

DDanGamble Are you able to currently export multiple workflows? This seems to work ``` [[w...

D

Diogo Ferreira•10/25/24, 11:42 AM

Thank you reporting it @DanGamble let us take a look at it

DDiogo Ferreira Thank you reporting it @DanGamble let us take a look at it

D

DanGamble•10/25/24, 11:43 AM

I was hoping to have ~10/15 or so. I can try and bundle them all into 1 for now as they are quite throwaway

DDanGamble Are you able to currently export multiple workflows? This seems to work ``` [[w...

M

Matt Silverlock•10/25/24, 12:02 PM

You can, yes. This looks like a local dev specific bug (which we can fix). This works:

export class FirstWorkflow extends WorkflowEntrypoint<Env, Params> {
    async run(event: WorkflowEvent<Params>, step: WorkflowStep) {
        const files = await step.do('my first step', async () => {
            return {
                inputParams: event,
                files: [],
            };
        });

        const apiResponse = await step.do('some other step', async () => {
            let resp = await fetch('https://api.cloudflare.com/client/v4/ips');
            return await resp.json<any>();
        });
    }
}

export class SecondWorkflow extends WorkflowEntrypoint<Env, Params> {
    async run(event: WorkflowEvent<Params>, step: WorkflowStep) {
        const files = await step.do('some step', async () => {
            return {};
        });

export class FirstWorkflow extends WorkflowEntrypoint<Env, Params> {
    async run(event: WorkflowEvent<Params>, step: WorkflowStep) {
        const files = await step.do('my first step', async () => {
            return {
                inputParams: event,
                files: [],
            };
        });

        const apiResponse = await step.do('some other step', async () => {
            let resp = await fetch('https://api.cloudflare.com/client/v4/ips');
            return await resp.json<any>();
        });
    }
}

export class SecondWorkflow extends WorkflowEntrypoint<Env, Params> {
    async run(event: WorkflowEvent<Params>, step: WorkflowStep) {
        const files = await step.do('some step', async () => {
            return {};
        });

export class FirstWorkflow extends WorkflowEntrypoint<Env, Params> {
    async run(event: WorkflowEvent<Params>, step: WorkflowStep) {
        const files = await step.do('my first step', async () => {
            return {
                inputParams: event,
                files: [],
            };
        });

        const apiResponse = await step.do('some other step', async () => {
            let resp = await fetch('https://api.cloudflare.com/client/v4/ips');
            return await resp.json<any>();
        });
    }
}

export class SecondWorkflow extends WorkflowEntrypoint<Env, Params> {
    async run(event: WorkflowEvent<Params>, step: WorkflowStep) {
        const files = await step.do('some step', async () => {
            return {};
        });

export class FirstWorkflow extends WorkflowEntrypoint<Env, Params> {
    async run(event: WorkflowEvent<Params>, step: WorkflowStep) {
        const files = await step.do('my first step', async () => {
            return {
                inputParams: event,
                files: [],
            };
        });

        const apiResponse = await step.do('some other step', async () => {
            let resp = await fetch('https://api.cloudflare.com/client/v4/ips');
            return await resp.json<any>();
        });
    }
}

export class SecondWorkflow extends WorkflowEntrypoint<Env, Params> {
    async run(event: WorkflowEvent<Params>, step: WorkflowStep) {
        const files = await step.do('some step', async () => {
            return {};
        });

[[workflows]]
name = "first-workflow"
binding = "FIRST_WORKFLOW"
class_name = "FirstWorkflow"

[[workflows]]
name = "second-workflow"
binding = "SECOND_WORKFLOW"
class_name = "SecondWorkflow"

[[workflows]]
name = "first-workflow"
binding = "FIRST_WORKFLOW"
class_name = "FirstWorkflow"

[[workflows]]
name = "second-workflow"
binding = "SECOND_WORKFLOW"
class_name = "SecondWorkflow"

[[workflows]]
name = "first-workflow"
binding = "FIRST_WORKFLOW"
class_name = "FirstWorkflow"

[[workflows]]
name = "second-workflow"
binding = "SECOND_WORKFLOW"
class_name = "SecondWorkflow"

[[workflows]]
name = "first-workflow"
binding = "FIRST_WORKFLOW"
class_name = "FirstWorkflow"

[[workflows]]
name = "second-workflow"
binding = "SECOND_WORKFLOW"
class_name = "SecondWorkflow"

➜  npx wrangler@latest deploy

Total Upload: 1.48 KiB / gzip: 0.62 KiB
Your worker has access to the following bindings:
- Workflows:
  - FIRST_WORKFLOW: FirstWorkflow
  - SECOND_WORKFLOW: SecondWorkflow
Uploaded two-workflows (1.76 sec)

➜  npx wrangler@latest deploy

Total Upload: 1.48 KiB / gzip: 0.62 KiB
Your worker has access to the following bindings:
- Workflows:
  - FIRST_WORKFLOW: FirstWorkflow
  - SECOND_WORKFLOW: SecondWorkflow
Uploaded two-workflows (1.76 sec)

➜  npx wrangler@latest deploy

Total Upload: 1.48 KiB / gzip: 0.62 KiB
Your worker has access to the following bindings:
- Workflows:
  - FIRST_WORKFLOW: FirstWorkflow
  - SECOND_WORKFLOW: SecondWorkflow
Uploaded two-workflows (1.76 sec)

➜  npx wrangler@latest deploy

Total Upload: 1.48 KiB / gzip: 0.62 KiB
Your worker has access to the following bindings:
- Workflows:
  - FIRST_WORKFLOW: FirstWorkflow
  - SECOND_WORKFLOW: SecondWorkflow
Uploaded two-workflows (1.76 sec)

Showing last 3 workflows:
┌───────────────────┬────────────────┬────────────────┬─────────────────────────┬─────────────────────────┐
│ Name              │ Script name    │ Class name     │ Created                 │ Modified                │
├───────────────────┼────────────────┼────────────────┼─────────────────────────┼─────────────────────────┤
│ first-workflow    │ two-workflows  │ FirstWorkflow  │ 10/25/2024, 12:01:24 PM │ 10/25/2024, 12:01:24 PM │
├───────────────────┼────────────────┼────────────────┼─────────────────────────┼─────────────────────────┤
│ second-workflow   │ two-workflows  │ SecondWorkflow │ 10/25/2024, 12:01:24 PM │ 10/25/2024, 12:01:24 PM │
├───────────────────┼────────────────┼────────────────┼─────────────────────────┼─────────────────────────┤
│ workflows-starter │ workflows-demo │ MyWorkflow     │ 10/23/2024, 11:33:58 AM │ 10/24/2024, 7:23:58 PM  │
└───────────────────┴────────────────┴────────────────┴─────────────────────────┴─────────────────────────┘

Showing last 3 workflows:
┌───────────────────┬────────────────┬────────────────┬─────────────────────────┬─────────────────────────┐
│ Name              │ Script name    │ Class name     │ Created                 │ Modified                │
├───────────────────┼────────────────┼────────────────┼─────────────────────────┼─────────────────────────┤
│ first-workflow    │ two-workflows  │ FirstWorkflow  │ 10/25/2024, 12:01:24 PM │ 10/25/2024, 12:01:24 PM │
├───────────────────┼────────────────┼────────────────┼─────────────────────────┼─────────────────────────┤
│ second-workflow   │ two-workflows  │ SecondWorkflow │ 10/25/2024, 12:01:24 PM │ 10/25/2024, 12:01:24 PM │
├───────────────────┼────────────────┼────────────────┼─────────────────────────┼─────────────────────────┤
│ workflows-starter │ workflows-demo │ MyWorkflow     │ 10/23/2024, 11:33:58 AM │ 10/24/2024, 7:23:58 PM  │
└───────────────────┴────────────────┴────────────────┴─────────────────────────┴─────────────────────────┘

Showing last 3 workflows:
┌───────────────────┬────────────────┬────────────────┬─────────────────────────┬─────────────────────────┐
│ Name              │ Script name    │ Class name     │ Created                 │ Modified                │
├───────────────────┼────────────────┼────────────────┼─────────────────────────┼─────────────────────────┤
│ first-workflow    │ two-workflows  │ FirstWorkflow  │ 10/25/2024, 12:01:24 PM │ 10/25/2024, 12:01:24 PM │
├───────────────────┼────────────────┼────────────────┼─────────────────────────┼─────────────────────────┤
│ second-workflow   │ two-workflows  │ SecondWorkflow │ 10/25/2024, 12:01:24 PM │ 10/25/2024, 12:01:24 PM │
├───────────────────┼────────────────┼────────────────┼─────────────────────────┼─────────────────────────┤
│ workflows-starter │ workflows-demo │ MyWorkflow     │ 10/23/2024, 11:33:58 AM │ 10/24/2024, 7:23:58 PM  │
└───────────────────┴────────────────┴────────────────┴─────────────────────────┴─────────────────────────┘

Showing last 3 workflows:
┌───────────────────┬────────────────┬────────────────┬─────────────────────────┬─────────────────────────┐
│ Name              │ Script name    │ Class name     │ Created                 │ Modified                │
├───────────────────┼────────────────┼────────────────┼─────────────────────────┼─────────────────────────┤
│ first-workflow    │ two-workflows  │ FirstWorkflow  │ 10/25/2024, 12:01:24 PM │ 10/25/2024, 12:01:24 PM │
├───────────────────┼────────────────┼────────────────┼─────────────────────────┼─────────────────────────┤
│ second-workflow   │ two-workflows  │ SecondWorkflow │ 10/25/2024, 12:01:24 PM │ 10/25/2024, 12:01:24 PM │
├───────────────────┼────────────────┼────────────────┼─────────────────────────┼─────────────────────────┤
│ workflows-starter │ workflows-demo │ MyWorkflow     │ 10/23/2024, 11:33:58 AM │ 10/24/2024, 7:23:58 PM  │
└───────────────────┴────────────────┴────────────────┴─────────────────────────┴─────────────────────────┘

MMatt Silverlock You can, yes. This looks like a local dev specific bug (which we can fix). This ...

D

DanGamble•10/25/24, 12:53 PM

Do you mean this works specifically in production or the

@latest

@latest

works with local dev too?

DDanGamble Do you mean this works specifically in production or the `@latest` works with lo...

D

Diogo Ferreira•10/25/24, 12:54 PM

It will work in production. The bug exists on local dev for the time being

DDanGamble Do you mean this works specifically in production or the `@latest` works with lo...

D

DanGamble•10/25/24, 12:54 PM

Just saw this was

deploy

deploy

so assume it still foobars in local dev, thanks @elithrar. Will do 1 big workflow for now as I really only need this to run locally as it's for data migrations/fixtures

D

DanGamble•10/25/24, 1:13 PM

Can you have a step within a step or does it all need to be top level?

M

makisuo•10/25/24, 1:17 PM

Just for clarification the 30sec of active CPU time is for the complete workflow and not per Step right?

DDanGamble Can you have a step within a step or does it all need to be top level?

M

Matt Silverlock•10/25/24, 1:35 PM

You can nest steps (try it!)

AAdvany A way around that would be to save it to r2/kv and then use in the next step?

D

dave•10/25/24, 1:37 PM

+1 to using the S3 API, ideally with aws4fetch.

I personally recommend considering Backblaze B2 to use within a Worker, since that way you can have caching on the

fetch

fetch

. (Sadly R2 caching doesn't work within a Worker.) You can also do a server-side mirror with B2 too for high availability, though now that I think about that, it doesn't matter as much with workflows (as they can retry).

MMatt Silverlock This is just early days: it's a beta 🙂

D

dave•10/25/24, 1:38 PM

pfff, that's not gonna stop me from shipping it in prod.

M

makrdev•10/25/24, 2:29 PM

Is there a mechanism to throttle functions that exceed the concurrency limit or we just loose them?

Mmakrdev Is there a mechanism to throttle functions that exceed the concurrency limit or ...

F

fry69•10/25/24, 2:35 PM

I think they just get queued? If they fail, they get also queued for retry? It should not be possible to actually fail hard or get lost unless you programmatically allow this.

C

Chaika•10/25/24, 2:36 PM

any hard retry count/timeout limits? Doesn't appear any are documented

CChaika any hard retry count/timeout limits? Doesn't appear any are documented

F

fry69•10/25/24, 2:37 PM

There are some limits of course -> https://developers.cloudflare.com/workflows/reference/limits/

Ffry69 There are some limits of course -> https://developers.cloudflare.com/workflows/r...

C

Chaika•10/25/24, 2:37 PM

Right, and retry limit/timeout aren't on there

Ffry69 I think they just get queued? If they fail, they get also queued for retry? It s...

D

Diogo Ferreira•10/25/24, 2:38 PM

Precisely this! When the concurrency limits are the reached, new Workflow Instances are queue, until new slots become available

CChaika Right, and retry limit/timeout aren't on there

F

fry69•10/25/24, 2:40 PM

Well, if your workflow constantly fails to do something it will reach the CPU execution time limit at some point

Ffry69 Well, if your workflow constantly fails to do something it will reach the CPU ex...

C

Chaika•10/25/24, 2:41 PM

that's a good point, 30s is a lot though

Original message was deleted

I

itsmatteomanf•10/25/24, 2:42 PM

according to this message though it seems like the 30s cpu time is per step...

CChaika that's a good point, 30s is a lot though

F

fry69•10/25/24, 2:43 PM

You can configure exponential backoff to limit harm -> https://developers.cloudflare.com/workflows/build/sleeping-and-retrying/#retry-steps

Iitsmatteomanf according to this message though it seems like the 30s cpu time is per step...

C

Chaika•10/25/24, 2:44 PM

now I have even more questions about if retries reset that or not lol

CChaika any hard retry count/timeout limits? Doesn't appear any are documented

D

Diogo Ferreira•10/25/24, 2:49 PM

No. You can go like,
{retries: { limit: 1000000, delay: '1 second', backoff: 'constant', }, timeout: '10 minutes'},

This refers to a step config

DDiogo Ferreira No. You can go like, {retries: { limit: 1000000, delay: '1 second',...

C

Chaika•10/25/24, 2:50 PM

Cool, no limits on the timeout part either? Could do '90 years'?
The cpu time is interesting too, Skye seemed to be saying above that it was per step? If so, how do retries work with that?

CChaika Cool, no limits on the timeout part either? Could do '90 years'? The cpu time is...

D

Diogo Ferreira•10/25/24, 2:52 PM

We never tried such a value

But yes, could do 90 years

F

fry69•10/25/24, 2:54 PM

timeout: 'until apocalypse'

CChaika now I have even more questions about if retries reset that or not lol

D

Diogo Ferreira•10/25/24, 2:54 PM

No, they don't reset. If one of a. retries reached or b. timeout reached, the step fails and the workflow will stop as errored

C

Chaika•10/25/24, 2:58 PM

Got it, thanks!

Exactly, I had an idea to make an abstraction on top of workflows to save every step to R2 and lazil

Similar Threads

Similar Threads

Similar Threads