I want to download a large file from r2 bucket via cloudflare worker, download starts but unlike goo

I want to download a large file from r2 bucket via cloudflare worker, download starts but unlike google drive export links it doesn't support threading/part downloads in IDM

ⱤⱤØ₥₳₦I want to download a large file from r2 bucket via cloudflare worker, download s...

Erisa•1/13/24, 5:31 PM

You need to implement ranged requests, heres an example worker and library that does https://github.com/kotx/render

EErisa You need to implement ranged requests, heres an example worker and library that ...

Ɽ

ⱤØ₥₳₦OP•1/13/24, 5:55 PM

Each range request will count as read?

Erisa•1/13/24, 5:56 PM

Yes thats how those threaded downloaders work, they split the download across multiple threads, each of which is a separate request

Erisa•1/13/24, 5:56 PM

Ranged requests are an important part of this because how it works is each thread downloads a different piece of the file

Erisa•1/13/24, 5:57 PM

e.g. 10 threads = 10 requests, each asking for a different 10% of the file

EErisa e.g. 10 threads = 10 requests, each asking for a different 10% of the file

Ɽ

ⱤØ₥₳₦OP•1/13/24, 5:57 PM

Gotcha, thanks for answering

Erisa•1/13/24, 5:57 PM

No problem

EErisa e.g. 10 threads = 10 requests, each asking for a different 10% of the file

Ɽ

ⱤØ₥₳₦OP•1/13/24, 6:00 PM

And what if i use url param for asking specific range data, then url will be cached as HIT

? Will that reduce requests unfathomably?

Erisa•1/13/24, 6:00 PM

Erisa•1/13/24, 6:01 PM

Range requests are separate requests yes, but I think there may have been a misunderstanding of how this works with the cache

Erisa•1/13/24, 6:01 PM

Once a file is in cache, you can throw infinite ranged requests at it and itll all be cache hits that dont touch R2. But it will still invoke your Worker if you use one

Erisa•1/13/24, 6:02 PM

Until the file falls out of cache, and then the ranges will go to R2 until the file is back in the cache

EErisa Once a file is in cache, you can throw infinite ranged requests at it and itll a...

Ɽ

ⱤØ₥₳₦OP•1/13/24, 6:03 PM

But i have files in gig size, and cloudflare won't cache individual one. So I'll try make it cache url param that only gives data for specific range

Erisa•1/13/24, 6:03 PM

Okay yeah, so if the file is too large to fit in cache on its own then thats true

Erisa•1/13/24, 6:04 PM

And if you managed to write a client implementation to do ranges with query parameters, yes that would treat them as separate files

Erisa•1/13/24, 6:04 PM

Enjoy

EErisa And if you managed to write a client implementation to do ranges with query para...

Ɽ

ⱤØ₥₳₦OP•1/13/24, 6:05 PM

Thank you very very much

you guys are always up for help

X•1/13/24, 7:02 PM

hello everyone

az•1/13/24, 9:51 PM

does early hints work with workers?

az•1/13/24, 9:51 PM

fetching a worker route with early hints enabled and not getting any

az•1/13/24, 9:53 PM

would there be any benefit anyways when all requests only hit the edge?

az•1/13/24, 9:54 PM

they have been reenabled a while ago i think

az•1/13/24, 9:55 PM

https://community.cloudflare.com/t/early-hints-and-encrypted-client-hello-ech-are-currently-disabled-globally/567730

az•1/13/24, 9:55 PM

are early hints only sent when a request to the origin is made?

az•1/13/24, 9:56 PM

if an edge cache hit (or worker hit) then the browser gets the entire page including any links from the edge

az•1/13/24, 9:57 PM

so even if an early hint is sent the client would get them at basically the same time as the rest of the page

az•1/13/24, 9:57 PM

no idea if that makes sense but thats my thought process lol

kian•1/13/24, 10:00 PM

Early Hints work fine w/ Workers

Kkian Early Hints work fine w/ Workers

az•1/13/24, 10:02 PM

any idea why i might not be seeing them?

kian•1/13/24, 10:03 PM

All of the reasons/requirements are on https://developers.cloudflare.com/cache/advanced-configuration/early-hints/

az•1/13/24, 10:05 PM

will take a look thanks

Darkardon•1/14/24, 12:22 AM

where do i go if i need help with deploy a server via wrangler

Darkardon•1/14/24, 12:22 AM

*deploying

DDarkardon where do i go if i need help with deploy a server via wrangler

Cyb3r-Jak3•1/14/24, 12:23 AM

You mean like a worker?

Darkardon•1/14/24, 12:30 AM

yes

F0rce•1/14/24, 12:32 AM

just out of curiosity, what could be the reasons for following error ?

GET https://murtleontour.ruxiom.dev/ - Exception Thrown @ 13.1.2024, 17:40:28
✘ [ERROR]   Error: The script will never generate a response.


GET https://murtleontour.ruxiom.dev/ - Ok @ 13.1.2024, 17:40:36

GET https://murtleontour.ruxiom.dev/ - Exception Thrown @ 13.1.2024, 17:40:28
✘ [ERROR]   Error: The script will never generate a response.


GET https://murtleontour.ruxiom.dev/ - Ok @ 13.1.2024, 17:40:36

The

The script will never generate a response.

The script will never generate a response.

error only happens super randomly is completely undebuggable and after 8 seconds everything seems to be working completely fine. It's just super hard and random to debug because the tail literally says nothing. And Sentry (which is integrated) also does not throw an error. So its literally "poking in nothing" and I cant figure it out. Does anybody have to deal with it aswell or any direction that I could be pointed ?

DDarkardon yes

Cyb3r-Jak3•1/14/24, 12:34 AM

#workers-help

FF0rce just out of curiosity, what could be the reasons for following error ? ``` GET ...

kian•1/14/24, 12:41 AM

Usually related to awaiting promises from other requests, i.e global scope stuff.

kian•1/14/24, 12:41 AM

Workers sees that Request A hasn't returned anything, yet Request A has no I/O left, so it can never finish.

kian•1/14/24, 12:41 AM

It has no insight that Request A is waiting on I/O from Request B, which is only possible via the global scope

Kkian Usually related to awaiting promises from other requests, i.e global scope stuff...

F0rce•1/14/24, 12:43 AM

as I am using storyblok (the cms) in an astro page (using ssr) it could be that the request to storyblok itself is failing ? should I try to improve the error handling when the request does not work ? or what could the correct approach be

F0rce•1/14/24, 12:49 AM

because thats the only "subrequest" / awaited request that I am aware of

F0rce•1/14/24, 12:50 AM

[...slug].astro

[...slug].astro

let story;
try {
  const { data } = await storyblokApi.get(
    `cdn/stories/${slug === undefined ? "startseite" : slug}`,
    {
      version:
        import.meta.env.PUBLIC_ENV === "production" ? "published" : "draft",
      resolve_links: "url",
    }
  );

  story = data.story;
} catch (error) {
  throw new Error(toErrorWithMessage(error).message);
}

let story;
try {
  const { data } = await storyblokApi.get(
    `cdn/stories/${slug === undefined ? "startseite" : slug}`,
    {
      version:
        import.meta.env.PUBLIC_ENV === "production" ? "published" : "draft",
      resolve_links: "url",
    }
  );

  story = data.story;
} catch (error) {
  throw new Error(toErrorWithMessage(error).message);
}

this is the only part where I await a request, ill see if that implementation does the job

itsnamanjain•1/14/24, 5:32 AM

Ɽ

ⱤØ₥₳₦OP•1/14/24, 9:20 AM

If I cache worker responses, and get HIT all the time, then will that count as worker request?

Erisa•1/14/24, 9:44 AM

Yes

GET https://murtleontour.ruxiom.dev/ - Exception Thrown @ 13.1.2024, 17:40:28 ✘ [ERROR] Error: The script will never generate a response. GET https://murtleontour.ruxiom.dev/ - Ok @ 13.1.2024, 17:40:36

let story; try { const { data } = await storyblokApi.get( `cdn/stories/${slug === undefined ? "startseite" : slug}`, { version: import.meta.env.PUBLIC_ENV === "production" ? "published" : "draft", resolve_links: "url", } ); story = data.story; } catch (error) { throw new Error(toErrorWithMessage(error).message); }

I want to download a large file from r2 bucket via cloudflare worker, download starts but unlike goo

Similar Threads

Similar Threads

Similar Threads