Hey Everyone! 👋 Building a Cloudflare-Powered File Sharing App – Seeking Feedback on Architecture

Hey Everyone!

Building a Cloudflare-Powered File Sharing App – Seeking Feedback on Architecture

I’m working on a file-sharing application where users can upload files and get a shareable link for others to download or upload additional files. My goal is to make it efficient, scalable, and cost-effective using Cloudflare’s ecosystem.

High-Level Architecture:
Cloudflare R2 stores uploaded files.
Event notifications send messages to a Cloudflare Queue when a file is uploaded.
A Cloudflare Worker consumes these messages and triggers a Cloudflare Workflow to handle file transfers.
The workflow:
Initiates a multipart upload to a cheaper storage bucket.
Reads the R2 object as a stream and uploads it in chunks.
Processes parts in bulk, each bulk handled in a separate step.
Sends a complete command if successful or an abort command if any step fails.
Questions I’d Love Feedback On:

How many workflows can run in parallel?

I trigger the workflow from a worker bound to a queue—how long can a workflow run?

What’s the best approach for handling uploads from the client?

whats counts for sub-requests in a workflow ?

Option 1: Upload to a worker endpoint (via Cloudflare Access) and trigger a workflow to handle multipart upload. Is this possible?
Option 2: Upload directly to R2 using se a PUT request for file under 50 MIB, for larger files use multipart upload:
(a) Temporary credentials (Access Key/Secret Key) scoped to the user's prefix (username).
(b) Pre-signed URLs (short expiration) for uploads/downloads.
(c) Worker with access for authenticated user with zero trust.

Would love to hear your thoughts! Looking forward to the discussion.

TTuna Hey Everyone! 👋 Building a Cloudflare-Powered File Sharing App – Seeking Feedba...

Victor•2/20/25, 8:40 PM

1 & 2 https://developers.cloudflare.com/workflows/reference/limits/
4 https://developers.cloudflare.com/workers/platform/limits/#subrequests

TunaOP•2/21/25, 8:44 AM

is a step in a workflow can be think of a single worker? does every step have it's own limitation of sub-request and open connection, cpu time, or the quota limitation is for the entire workflow ? its very unclear from the documentation.

TTuna is a step in a workflow can be think of a single worker? does every step have it...

Seekerdasbatatas•2/21/25, 11:20 AM

Sub-request and open connection limits are per workflow instance. We automatically retry when you reach CPU time and memory limits, but it's not really in a per-step basis (because you can have steps in parallel)

TunaOP•2/21/25, 2:04 PM

@Luís Duarte

and what is an instance of a workflow ?

TTuna @Luís Duarte 🧑🍳 and what is an instance of a workflow ?

Isaac McFadyen•2/21/25, 2:05 PM

A single run of the workflow.

Isaac McFadyen•2/21/25, 2:06 PM

I.e. you run the workflow, that's 1 instance, you run it again, that's 2 instances.

Isaac McFadyen•2/21/25, 2:06 PM

The CPU time and memory may be shared between runs (the same with any Worker) but the subrequests and concurrent requests are per run.

TunaOP•2/21/25, 2:19 PM

Sub-request and open connection limits are per workflow instance?
workflow limits are the same of a worker? cause Im looking for sub-requests and open connections here
https://developers.cloudflare.com/workflows/reference/limits/
and i can't find anything explicit

Cloudflare Docs

Limits · Cloudflare Workflows docs

Limits that apply to authoring, deploying, and running Workflows are detailed below.

Seekerdasbatatas•2/21/25, 2:37 PM

subrequest and open connection limits are inherited from the worker limits

TunaOP•2/21/25, 4:07 PM

@Seekerdasbatatas if a step retries its counts for another round of this step subrequests from the workflow quota ? :NotLikeThis:

TunaOP•2/21/25, 4:13 PM

im trying to find how to split the file into multipart without hiting the limit

TunaOP•2/21/25, 4:18 PM

step 1: init multipart upload command
step 2: creating an array of steps, each step does. promise.all([promises of uploading a part]) -> maximum 2 concurent uploads
step 3. complete request
if failed abort reqeust

TTuna im trying to find how to split the file into multipart without hiting the limit

Seekerdasbatatas•2/21/25, 5:59 PM

if you mean R2's multipart upload, you could calculate the part size by dividing the total file size by 990 sub-requests (if you have the file length, i also don't do a 1000 to give you some leeway for other stuff ): https://developers.cloudflare.com/r2/objects/multipart-objects/

Cloudflare Docs

Multipart upload · Cloudflare R2 docs

R2 supports S3 API's Multipart Upload with some limitations.

Seekerdasbatatas•2/21/25, 6:00 PM

the maximum is 5GiB so using this method you could upload 5TiB (ish)

fforres•2/22/25, 7:14 PM

Hey folks, been using workflows is a lot more lately, mostly as encapsulation of agentic flows.

Wondering if there are either plans or existing strategies to visualize AI Gateway calls/entries related to workflow runs?

(same w/ sub-request or triggering sub-workflows?)

So far, thinking of just adding some metadata to the LLM calls... and then manual filter, but was thinking as you folks have control over the gateway + knowldge of the workflow/worker Instances there was something better.

(Would be awesome to be able to shove metadata/custom data to workflow metrics)

Ffforres Hey folks, been using workflows is a lot more lately, mostly as encapsulation of...

Matt Silverlock•2/23/25, 12:38 AM

What kind of metadata?

TunaOP•2/23/25, 7:09 AM

Hey everyone what is the road map of the workflows? is it something i can start using on production ?

TTuna Hey everyone what is the road map of the workflows? is it something i can start ...

tadhglewis•2/23/25, 7:15 AM

It's not public. It's beta so inherently no, it's not business critical ready. Yes some people probably use it for prod systems, depends on risk appetite

SSeekerdasbatatas if you mean R2's multipart upload, you could calculate the part size by dividing...

TunaOP•2/23/25, 7:18 AM

Im using one workflow to for multipart upload, and with current limitation the largest file i can support is 15 GiB and not 5TiB file.
taking into consideration 15 minutes time limit, and if every upload part in worst case will retry.
can you take a look at my calculator https://rb.gy/tdpydr, maybe im wrong ? i have seen a lot of question regards limits, it would be nice to create a helper so developers can use this to manage their quota limits.

Free link shortener: Fast & simple short URLs

Need to shorten a URL? Use RB.GY for free, memorable shortened links that are easy to track. Ideal for social media, emails, and ads.

Ttadhglewis It's not public. It's beta so inherently no, it's not business critical ready. Y...

TunaOP•2/23/25, 7:24 AM

Thanks for the quick answer!

My use case is to move every uploaded file to R2 to another storage provider. For files larger than 100 MiB, I’m using the workflow for multipart upload (start multipart, upload parts, complete, or abort on failure).

Do you think the workflow beta version can handle this in production at 100 TiB of files per day, or am I way off? If this beta isn’t suited for such scale, is there another Cloudflare service you’d recommend that could handle this load and make it easier to transition my architecture from the workflow?

TTuna Thanks for the quick answer! 😊 My use case is to move every uploaded file to R...

Matt Silverlock•2/23/25, 9:38 PM

For clarity: it's not the volume of data, it's that a beta service might experience the occasional disruption or bug as we continue to iterate on it. We have teams and other customers using Workflows in production, but we ask folks to decide for themselves here.

steyblind•2/24/25, 1:12 AM

Hi all, just started using workflows. Overall, extremely rad.

Something baffling me is the __LOCAL_DEV_STEP_OUTPUTS and what the equivalent would be from a deployed workflow? obvs I can trial-and-error this, but I wanted to ID documentation on this vs. "production" outputs

MMatt Silverlock What kind of metadata?

fforres•2/24/25, 5:47 AM

Oh, like attaching the workflow-id to the

cf-aig-metadata

cf-aig-metadata

And then filter in gateway view for it.

ANBU•2/24/25, 9:15 AM

Hey, I'm working on a next js app that is served via workers (not pages) with open nextjs
Now, in the same app, I'm trying to use a workflow but getting this error

Worker "workflows:test"'s binding "USER_WORKFLOW" refers to service "core:user:" with a named entrypoint "TestWorkflow", but "core:user:" has no such named entrypoint.

Worker "workflows:test"'s binding "USER_WORKFLOW" refers to service "core:user:" with a named entrypoint "TestWorkflow", but "core:user:" has no such named entrypoint.

I can see some mentions of this error in this channel, but they are related to pages deployment, which I'm not using.
Any tips on how to move forward?

AANBU Hey, I'm working on a next js app that is served via workers (not pages) with op...

Matt Silverlock•2/24/25, 10:57 AM

Can you show your code? Are you exporting the Workflow definition?

MMatt Silverlock Can you show your code? Are you exporting the Workflow definition?

ANBU•2/24/25, 4:01 PM

I'm using the code mentioned here https://developers.cloudflare.com/workflows/get-started/cli-quick-start/

This is what my directory structure looks like

.
├── src
│   ├── app (next js app)
│   │   ├── api
│   │   │   └── [[...route]]
│   │   │       └── route.ts
│   │   ├── page.tsx
│   │   └── layout.tsx
│   └── server (hono server)
│       ├── routes
│       │   └── workflow-route.ts
│       ├── workflows
│       │   └── my-workflow.ts
│       └── _app.ts 
├── wrangler.jsonc
└── package.json

.
├── src
│   ├── app (next js app)
│   │   ├── api
│   │   │   └── [[...route]]
│   │   │       └── route.ts
│   │   ├── page.tsx
│   │   └── layout.tsx
│   └── server (hono server)
│       ├── routes
│       │   └── workflow-route.ts
│       ├── workflows
│       │   └── my-workflow.ts
│       └── _app.ts 
├── wrangler.jsonc
└── package.json

Cloudflare Docs

CLI quick start · Cloudflare Workflows docs

Workflows allow you to build durable, multi-step applications using the Workers platform. A Workflow can automatically retry, persist state, run for hours or days, and coordinate between third-party APIs.

ANBU•2/24/25, 4:04 PM

I'm exporting the workflow in my-workflow.ts

ANBU•2/24/25, 4:23 PM

I also tinkered with the given example on a separate repo created using

npm create cloudflare@latest workflows-starter -- --template "cloudflare/workflows-starter"

npm create cloudflare@latest workflows-starter -- --template "cloudflare/workflows-starter"

and noticed that, even if I define the workflow in other file, I still need to export it from index.ts, I think this is somewhat related, but not able to figure this out in next js

AANBU I also tinkered with the given example on a separate repo created using `npm cre...

Matt Silverlock•2/24/25, 4:57 PM

TestWorkflow

TestWorkflow

isn't from the example (from your error message) so I suspect you changed a binding or the Workflow name itself?

Matt Silverlock•2/24/25, 4:57 PM

with a named entrypoint "TestWorkflow"

with a named entrypoint "TestWorkflow"

If you didn't follow the guide as-is, it won't work

MMatt Silverlock `TestWorkflow` isn't from the example (from your error message) so I suspect you...

ANBU•2/24/25, 5:10 PM

Yup I changed the name in the bindings and code but rest all is same

ANBU•2/24/25, 5:15 PM

Just to be extra sure, created another workflow with the same name and bindings
Still getting the same error with MyWorkflow

ANBU•2/24/25, 5:16 PM

ANBU•2/24/25, 5:18 PM

AANBU Click to see attachment

Matt Silverlock•2/24/25, 5:51 PM

@Luís Duarte

@Diogo Ferreira - this one seems off?

dan•2/24/25, 7:07 PM

I see this message: workflows
but didn't see a response to it - is there a way to listen to workflow state via a stream?

AANBU Hey, I'm working on a next js app that is served via workers (not pages) with op...

Seekerdasbatatas•2/24/25, 7:37 PM

hey AFAICT, OpenNext uses their own entrypoint shim to be able to interoperate with Next: https://github.com/opennextjs/opennextjs-cloudflare/blob/main/packages/cloudflare/src/cli/templates/worker.ts

you will probably need to extend this worker template to export your workflow in the worker entrypoint file

GitHub

opennextjs-cloudflare/packages/cloudflare/src/cli/templates/worker....

Open Next.js adapter for Cloudflare. Contribute to opennextjs/opennextjs-cloudflare development by creating an account on GitHub.

Seekerdasbatatas•2/24/25, 7:38 PM

(this also applies to DOs, for example)

Seekerdasbatatas•2/24/25, 7:55 PM

or you can create another script/worker that just exports the workflow directly

Ddan I see this message: https://discord.com/channels/595317990191398933/129692370779...

Matt Silverlock•2/25/25, 11:32 AM

Not at the moment.

SSeekerdasbatatas or you can create another script/worker that just exports the workflow directly

ANBU•2/25/25, 12:21 PM

Sorry, I didn't get this one, it sounds simpler. How can this be achieved?

SSeekerdasbatatas hey AFAICT, OpenNext uses their own entrypoint shim to be able to interoperate w...

ANBU•2/25/25, 1:39 PM

Tried this by copying this file, exporting my workflow from there and changing main file in wrangler.jsonc
But still getting the same error
Is there anything else thats needed apart from this?

MMatt Silverlock Not at the moment.

dan•2/25/25, 4:28 PM

It'd be cool to see that at some point - it's a blocking feature for me to use this for my AI workflows

WILDE•2/25/25, 4:53 PM

Hi there! I am trying to replace some queues with workflows and I've realized that there is a known issue

Workflows does not support npx wrangler dev --remote.

Workflows does not support npx wrangler dev --remote.

I'd like to know if this is something that will be fixed in the future or will remain as it is. I have some workers that are connected to some external services such as clickhouse, upstash and some others. Build them locally with Docker for

yarn dev

yarn dev

to work will be crazy

ajay | dreamlit.ai•2/26/25, 1:37 AM

Hi, big fan of workflows, excited to start using it in our app.

Quick question, whenever Wrangler detects a change, it hot reloads the server, which kills any currently running workflows. Is there any way to either 1) keep them alive or 2) have some kind of startup script/hook that runs so we can run relaunch them ourselves?

DanGamble•2/26/25, 9:47 AM

abstract class BaseWorkflow extends WorkflowEntrypoint<Env, Params> {
  async run(event: WorkflowEvent<Params>, step: WorkflowStep) {
    const something = await step.do("do something", async () => {
      // ...
    })

    await this.execute(event, step)
  }

  abstract execute(event: WorkflowEvent<Params>, step: WorkflowStep): Promise<void>
}

class TestWorkflow extends BaseWorkflow {
  async execute(event: WorkflowEvent<Params>, step: WorkflowStep) {
    const something = await step.do("do something", async () => {
      // ...
    })
    
    await someStepsIWantToShare(event, step)
  }
}

async function someStepsIWantToShare (event: WorkflowEvent<Params>, step: WorkflowStep) {
  const anotherStep = await step.do("do something else", async () => {
    // ...
  })
}

abstract class BaseWorkflow extends WorkflowEntrypoint<Env, Params> {
  async run(event: WorkflowEvent<Params>, step: WorkflowStep) {
    const something = await step.do("do something", async () => {
      // ...
    })

    await this.execute(event, step)
  }

  abstract execute(event: WorkflowEvent<Params>, step: WorkflowStep): Promise<void>
}

class TestWorkflow extends BaseWorkflow {
  async execute(event: WorkflowEvent<Params>, step: WorkflowStep) {
    const something = await step.do("do something", async () => {
      // ...
    })
    
    await someStepsIWantToShare(event, step)
  }
}

async function someStepsIWantToShare (event: WorkflowEvent<Params>, step: WorkflowStep) {
  const anotherStep = await step.do("do something else", async () => {
    // ...
  })
}

Is this a valid way for work with Workflows?

Matt Silverlock•2/26/25, 7:41 PM

FYI: we've just raised the # of concurrent Workflows from ~100 to ~1000 (briefly) to (now) ~4,500 per Workflow definition: https://x.com/chatsidhartha/status/1894834426552459433

(Plans are to get to near ~10k or more per Workflow as we move forward)

Sid (@chatsidhartha) on X

Yesterday, we shipped an (minor) increase in concurrency limits for @CloudflareDev Workflows to 1,500 (from 100)

Today, we shipped a major improvement and concurrency limits are now up (AGAIN) to 4,500!

Happy building

https://t.co/kKXv4nPVtl

•

2/26/25, 7:38 PM

ajay | dreamlit.ai•2/26/25, 8:39 PM

What happens when there's an active workflow running and there's a new version deployed? My understanding is that it will continue to run on the older version... but what if it invokes another workflow via its service bindings? Will that run in the old version or the new?

Aajay | dreamlit.ai What happens when there's an active workflow running and there's a new version d...

Matt Silverlock•2/26/25, 9:15 PM

Any running Workflow runs as-is today. Looking at options to provide config for this (e.g. terminate/restart all old instances on a deploy).

If it invokes another Workflow via a binding, that is still going to be the "latest" version. The old script isn't kept around. Workflows is no different here - if Worker A calls Worker B over a Service Binding, and you re-deploy Worker B, Worker A will always (only) call the latest code.

. ├── src │ ├── app (next js app) │ │ ├── api │ │ │ └── [[...route]] │ │ │ └── route.ts │ │ ├── page.tsx │ │ └── layout.tsx │ └── server (hono server) │ ├── routes │ │ └── workflow-route.ts │ ├── workflows │ │ └── my-workflow.ts │ └── _app.ts ├── wrangler.jsonc └── package.json

abstract class BaseWorkflow extends WorkflowEntrypoint<Env, Params> { async run(event: WorkflowEvent<Params>, step: WorkflowStep) { const something = await step.do("do something", async () => { // ... }) await this.execute(event, step) } abstract execute(event: WorkflowEvent<Params>, step: WorkflowStep): Promise<void> } class TestWorkflow extends BaseWorkflow { async execute(event: WorkflowEvent<Params>, step: WorkflowStep) { const something = await step.do("do something", async () => { // ... }) await someStepsIWantToShare(event, step) } } async function someStepsIWantToShare (event: WorkflowEvent<Params>, step: WorkflowStep) { const anotherStep = await step.do("do something else", async () => { // ... }) }

Hey Everyone! 👋 Building a Cloudflare-Powered File Sharing App – Seeking Feedback on Architecture

Similar Threads

Similar Threads

Similar Threads