Theres this which says 1200 req per 5 minutes, but I dont think thats the one youre running into? <h

Theres this which says 1200 req per 5 minutes, but I dont think thats the one youre running into? https://developers.cloudflare.com/fundamentals/api/reference/limits/

EErisa Theres this which says 1200 req per 5 minutes, but I dont think thats the one yo...

meyer•3/15/23, 12:41 AM

yeah i don’t think that’s the one i’m getting

Cyb3r-Jak3•3/15/23, 12:45 AM

I thought it would be. Wrangler just uses the normal API when uploading in bulk

meyer•3/15/23, 12:52 AM

innteresting… maybe it is? i’m definitely not sending 1200 requests in five minutes tho. 64 calls containing 2000 items each.

James•3/15/23, 1:06 AM

It's not uncommon for KV to throw 429s with bulk data unfortunately, even outside of the API req limits

meyer•3/15/23, 1:29 AM

oh dear. good to know, thank you. i’ll add some extra handling on my end.

Mmeyer oh dear. good to know, thank you. i’ll add some extra handling on my end.

beelzabub•3/15/23, 1:53 AM

64 concurrent calls with 2k items each? Yeah, that’s >100k writes/s if my math is right. That’s too high right now. I wouldn’t recommend going higher than 10k/s right now. KV2 will give us more breathing room. I do want to get to much higher later this year but it’ll depend on internal arch stuff.

Bbeelzabub 64 concurrent calls with 2k items each? Yeah, that’s >100k writes/s if my math i...

meyer•3/15/23, 2:04 AM

ahhhhhhh that makes sense, i wasn’t thinking of writes, just requests

Oleg Isonen•3/15/23, 4:04 PM

Hey folks, I am seeing something (strange?) - KV has much higher latency than R2, is that normal?

To back it up, here is what I am looking at.

For static JS, we are using workers for platforms and we use kv-asset-handler
For fonts, we use fetch from R2

Fonts are waiting for response under 50ms consistently
Scripts are waiting for response from 150ms to 500ms

We tried multiple regions, I am personally Berlin.

beelzabub•3/15/23, 4:08 PM

KV should not have higher latency than R2 generally. R2 can have better latency if you happen to be in the same region you placed the bucket (which is generally the default if the machine you created the bucket is the same machine making these requests). Even still, for an asset you fetch repeatedly KV should still be faster.

beelzabub•3/15/23, 4:10 PM

If you're changing the UUID of the script frequently, that would trigger a cold read every time. So this is very dependent on your test experiment environment.

Bbeelzabub KV should not have higher latency than R2 generally. R2 can have better latency ...

Oleg Isonen•3/15/23, 4:20 PM

We are not changing the UUID of the script during the tests, so its really static. https://webstudio.is

beelzabub•3/15/23, 4:35 PM

Ok. Let me take a look

beelzabub•3/15/23, 4:38 PM

So just so I'm clear, for example https://webstudio.is/build/entry.client-QHBMJZIF.js is taking 137ms in your screenshot and that's served from KV?

Bbeelzabub So just so I'm clear, for example https://webstudio.is/build/entry.client-QHBMJZ...

Oleg Isonen•3/15/23, 4:49 PM

yes!

beelzabub•3/15/23, 5:02 PM

So looking at our backend, it looks like there's no KV asset for that file. It looks like you might be trying to put cache in front of KV but then forgetting to update KV on a miss?

Ivan Starkov•3/15/23, 5:13 PM

Its worker for platform, it has following code.

      // @todo the whole logic looks strange, add test, write what it solves
      if (asset) {
        finalRequest = new Request(
          url.origin + "/" + asset,
          request as RequestInit
        );
      }

      const response = await getAssetFromKV(
        {
          request: finalRequest,
          waitUntil(promise) {
            return ctx.waitUntil(promise);
          },
        },
        {
          ASSET_NAMESPACE: env.__STATIC_CONTENT,
          ASSET_MANIFEST: assetManifest,
          cacheControl: {
            browserTTL: ttl,
            edgeTTL: ttl,
          },
        }
      );

      return response;

      // @todo the whole logic looks strange, add test, write what it solves
      if (asset) {
        finalRequest = new Request(
          url.origin + "/" + asset,
          request as RequestInit
        );
      }

      const response = await getAssetFromKV(
        {
          request: finalRequest,
          waitUntil(promise) {
            return ctx.waitUntil(promise);
          },
        },
        {
          ASSET_NAMESPACE: env.__STATIC_CONTENT,
          ASSET_MANIFEST: assetManifest,
          cacheControl: {
            browserTTL: ttl,
            edgeTTL: ttl,
          },
        }
      );

      return response;

KV is created by using

[site]
bucket = "./public"

[site]
bucket = "./public"

section in toml file
and exists and filled

4453161ad26e32a617c6c6d4b837c9d9/workers/kv/namespaces/989796259c334124b2e868bd9a0c3849

4453161ad26e32a617c6c6d4b837c9d9/workers/kv/namespaces/989796259c334124b2e868bd9a0c3849

Ivan Starkov•3/15/23, 5:15 PM

This library https://github.com/cloudflare/kv-asset-handler std code is used for static assets hosting

Ivan Starkov•3/15/23, 5:17 PM

timings are very unstable and sometimes can be very good

Ivan Starkov•3/15/23, 5:17 PM

But next reload

Ivan Starkov•3/15/23, 5:17 PM

Ivan Starkov•3/15/23, 5:17 PM

Font timing (r2 with cache rule) is independent

beelzabub•3/15/23, 5:18 PM

That library seems to have a lot of magic and I've never looked at it before

Oleg Isonen•3/15/23, 5:20 PM

The weird thing is that R2 has consistently good timing. Should we put everything in R2 then?

beelzabub•3/15/23, 5:20 PM

No. I'd recommend using KV properly

Ivan Starkov•3/15/23, 5:21 PM

Can it be because of worker is blocked by some heavy script, like first request is coming and cause some heavy job on worker - really 100-200ms computation, and consequent requests are waiting for worker finish that task?

beelzabub•3/15/23, 5:21 PM

The problem is that that kv-asset-handler is doing a lot of complicated things more than just hitting KV

beelzabub•3/15/23, 5:21 PM

It's using cache in front of KV and trying to honor caching headers from the origin afaict.

beelzabub•3/15/23, 5:22 PM

Any one of those things could be tripping you up in theory

beelzabub•3/15/23, 5:22 PM

Try disabling shouldEdgeCacheshouldEdgeCache in your options and see if that fixes it

Ivan Starkov•3/15/23, 5:23 PM

Trying

beelzabub•3/15/23, 5:23 PM

I think you do that by eliding ASSET_MANIFEST

beelzabub•3/15/23, 5:24 PM

(i.e. setting it to undefined)

Ivan Starkov•3/15/23, 5:27 PM

Seems like this https://github.com/cloudflare/kv-asset-handler/blob/c198236dc7444f99acad1f8ccffa2532b8fda879/src/index.ts#L227 i.e. bypassCache also set shouldEdgeCache=false

beelzabub•3/15/23, 5:28 PM

Yeah maybe

beelzabub•3/15/23, 5:30 PM

@cherryjimbo might know more about the specifics of this library as he seems to be the primary maintainer.

Ivan Starkov•3/15/23, 5:38 PM

Tried with cache bypass, seems like become worse but hard to say. Like 30/60ms worse

beelzabub•3/15/23, 5:39 PM

Can you try without that library at all?

Ivan Starkov•3/15/23, 5:41 PM

For me it looks like highlu correlated with 2nd request we have

Ivan Starkov•3/15/23, 5:41 PM

If it came from cache everything else is fast, KV etc

beelzabub•3/15/23, 5:41 PM

That library is doing too many fancy things. If KV is your source of truth, you can just read from it and set whatever cache control headers you need (e.g. you can store it within the KV metadata if you need to customize).

Ivan Starkov•3/15/23, 5:41 PM

If it calculated (and its really heavy on calulation) everything from KV is slow

James•3/15/23, 5:43 PM

Sorry I’m on mobile so can’t help too much at the moment, but kv-asset-handler shouldn’t be that far off normal KV performance. It uses

cache

cache

manually though

Ivan Starkov•3/15/23, 5:56 PM

I disabled heavy script and have same results for all requests

Ivan Starkov•3/15/23, 5:56 PM

beelzabub•3/15/23, 5:57 PM

What do you mean by "heavy script"?

Ivan Starkov•3/15/23, 5:57 PM

So it looks like we have single worker which does not process KV until heavy work on requested script

Ivan Starkov•3/15/23, 5:57 PM

Heavy - heavy in computation like 200ms to compute response

Ivan Starkov•3/15/23, 5:58 PM

?css-hash?css-hash in screenshots above

beelzabub•3/15/23, 5:59 PM

?css-hash in the last screenshot shows 41ms

// @todo the whole logic looks strange, add test, write what it solves if (asset) { finalRequest = new Request( url.origin + "/" + asset, request as RequestInit ); } const response = await getAssetFromKV( { request: finalRequest, waitUntil(promise) { return ctx.waitUntil(promise); }, }, { ASSET_NAMESPACE: env.__STATIC_CONTENT, ASSET_MANIFEST: assetManifest, cacheControl: { browserTTL: ttl, edgeTTL: ttl, }, } ); return response;

Theres this which says 1200 req per 5 minutes, but I dont think thats the one youre running into? <h

Similar Threads

Similar Threads

Similar Threads