5 MiB paid plan 🙂

B

beelzabub•3/10/23, 6:39 PM

Grrrr. Should have just tried it. That's what happens when you rely on information from 2 years ago. I actually helped plan the work needed to raise that limit. I just rolled off to start R2 before that work completed and I did a bad job of following up. Good to hear it's been raised

K

Kirk Ouimet•3/10/23, 11:58 PM

Hi team - is it possible to do a list operation on a KV and have it stop at a certain character? E.g. I have these entries

1 - key1/keyA
2 - key1/keyA/keyAA
3 - key1/keyA/keyAA/keyAAA

and I just want to list and get 1 and 2 but not 3

J

James•3/10/23, 11:59 PM

not to my knowledge

K

Kirk Ouimet•3/11/23, 12:01 AM

yeah kind of tricky trying to get a graph type situation going on using KV alone

B

beelzabub•3/11/23, 1:35 AM

I do want to get to make list operations more like R2, but I'm not sure we'll be able to do that this year.

B

beelzabub•3/11/23, 1:35 AM

Strong candidate for 2024 though.

B

beelzabub•3/11/23, 1:36 AM

I know that's not a great answer though in the short term.

Bbeelzabub I do want to get to make list operations more like R2, but I'm not sure we'll be...

K

Kirk Ouimet•3/11/23, 4:09 AM

Ah could you give me an example of the syntax that would be used to accomplish my goal? BTW I am building a new company on cloudflare and you guys are going to destroy every cloud provider this decade

KKirk Ouimet Ah could you give me an example of the syntax that would be used to accomplish m...

S

Sly•3/11/23, 11:37 AM

I’m just waiting for CF Auth and CF Mail

SSly I’m just waiting for CF Auth and CF Mail 😄

A

Advany•3/11/23, 1:28 PM

Cf mail is mailchannels?

E

Erisa•3/11/23, 1:48 PM

and #email-routing for the other direction

KKirk Ouimet Ah could you give me an example of the syntax that would be used to accomplish m...

B

beelzabub•3/12/23, 12:23 AM

There’s no way to do it from your worker today. I would imagine we’d unify it similar with how list works on R2

KKevH I'm looking to perform a list of keys in a namespace using the nodejs sdk. Looki...

C

Cyb3r-Jak3•3/14/23, 5:35 PM

You are correct. The node library is very out of date. The last commit was a year ago and the last tag two years ago

B

beelzabub•3/14/23, 6:45 PM

Wow I didn't even realize that was a thing. I'll raise it internally to see what the plans are for that.

KKevH Is it recommended to just use the api directly for now?

C

Cyb3r-Jak3•3/14/23, 6:54 PM

Yeah. The go api is the most update if you know go

Bbeelzabub Wow I didn't even realize that was a thing. I'll raise it internally to see what...

C

Cyb3r-Jak3•3/14/23, 6:55 PM

IIRC, there is a plan to auto generate clients once the shift to openapi is done. The node, python, and PHP libraries don’t receive a lot of love otherwise

M

meyer•3/14/23, 10:04 PM

can bulk KV write operations occur simultaneously? i'm writing around 128K items to a KV store in batches of 2000 over a period of about a minute and i'm noticing that some values aren't getting written.

M

meyer•3/14/23, 10:04 PM

at the moment i'm doing some further debugging to make sure that it's not me doing something dumb, but i figured i'd check here as well.

Mmeyer can bulk KV write operations occur simultaneously? i'm writing around 128K items...

M

meyer•3/15/23, 12:14 AM

mystery solved: 429 Too Many Requests

M

meyer•3/15/23, 12:15 AM

is rate limiting per IP or per API key?

J

James•3/15/23, 12:21 AM

Account ID in general, I believe

J

James•3/15/23, 12:22 AM

429s are pretty common when doing bulk stuff with KV

C

Cyb3r-Jak3•3/15/23, 12:22 AM

Yeah API rate limit is per account

M

meyer•3/15/23, 12:34 AM

ty

M

meyer•3/15/23, 12:35 AM

is rate limit info posted anywhere? googling for anything cloudflare-related + 429 gives me a whole lot of useless results

E

Erisa•3/15/23, 12:36 AM

Theres this which says 1200 req per 5 minutes, but I dont think thats the one youre running into? https://developers.cloudflare.com/fundamentals/api/reference/limits/

EErisa Theres this which says 1200 req per 5 minutes, but I dont think thats the one yo...

M

meyer•3/15/23, 12:41 AM

yeah i don’t think that’s the one i’m getting

C

Cyb3r-Jak3•3/15/23, 12:45 AM

I thought it would be. Wrangler just uses the normal API when uploading in bulk

M

meyer•3/15/23, 12:52 AM

innteresting… maybe it is? i’m definitely not sending 1200 requests in five minutes tho. 64 calls containing 2000 items each.

J

James•3/15/23, 1:06 AM

It's not uncommon for KV to throw 429s with bulk data unfortunately, even outside of the API req limits

M

meyer•3/15/23, 1:29 AM

oh dear. good to know, thank you. i’ll add some extra handling on my end.

Mmeyer oh dear. good to know, thank you. i’ll add some extra handling on my end.

B

beelzabub•3/15/23, 1:53 AM

64 concurrent calls with 2k items each? Yeah, that’s >100k writes/s if my math is right. That’s too high right now. I wouldn’t recommend going higher than 10k/s right now. KV2 will give us more breathing room. I do want to get to much higher later this year but it’ll depend on internal arch stuff.

Bbeelzabub 64 concurrent calls with 2k items each? Yeah, that’s >100k writes/s if my math i...

M

meyer•3/15/23, 2:04 AM

ahhhhhhh that makes sense, i wasn’t thinking of writes, just requests

O

Oleg Isonen•3/15/23, 4:04 PM

Hey folks, I am seeing something (strange?) - KV has much higher latency than R2, is that normal?

To back it up, here is what I am looking at.

For static JS, we are using workers for platforms and we use kv-asset-handler
For fonts, we use fetch from R2

Fonts are waiting for response under 50ms consistently
Scripts are waiting for response from 150ms to 500ms

We tried multiple regions, I am personally Berlin.

B

beelzabub•3/15/23, 4:08 PM

KV should not have higher latency than R2 generally. R2 can have better latency if you happen to be in the same region you placed the bucket (which is generally the default if the machine you created the bucket is the same machine making these requests). Even still, for an asset you fetch repeatedly KV should still be faster.

B

beelzabub•3/15/23, 4:10 PM

If you're changing the UUID of the script frequently, that would trigger a cold read every time. So this is very dependent on your test experiment environment.

Bbeelzabub KV should not have higher latency than R2 generally. R2 can have better latency ...

O

Oleg Isonen•3/15/23, 4:20 PM

We are not changing the UUID of the script during the tests, so its really static. https://webstudio.is

B

beelzabub•3/15/23, 4:35 PM

Ok. Let me take a look

B

beelzabub•3/15/23, 4:38 PM

So just so I'm clear, for example https://webstudio.is/build/entry.client-QHBMJZIF.js is taking 137ms in your screenshot and that's served from KV?

Bbeelzabub So just so I'm clear, for example https://webstudio.is/build/entry.client-QHBMJZ...

O

Oleg Isonen•3/15/23, 4:49 PM

yes!

B

beelzabub•3/15/23, 5:02 PM

So looking at our backend, it looks like there's no KV asset for that file. It looks like you might be trying to put cache in front of KV but then forgetting to update KV on a miss?

I

Ivan Starkov•3/15/23, 5:13 PM

Its worker for platform, it has following code.

      // @todo the whole logic looks strange, add test, write what it solves
      if (asset) {
        finalRequest = new Request(
          url.origin + "/" + asset,
          request as RequestInit
        );
      }

      const response = await getAssetFromKV(
        {
          request: finalRequest,
          waitUntil(promise) {
            return ctx.waitUntil(promise);
          },
        },
        {
          ASSET_NAMESPACE: env.__STATIC_CONTENT,
          ASSET_MANIFEST: assetManifest,
          cacheControl: {
            browserTTL: ttl,
            edgeTTL: ttl,
          },
        }
      );

      return response;

      // @todo the whole logic looks strange, add test, write what it solves
      if (asset) {
        finalRequest = new Request(
          url.origin + "/" + asset,
          request as RequestInit
        );
      }

      const response = await getAssetFromKV(
        {
          request: finalRequest,
          waitUntil(promise) {
            return ctx.waitUntil(promise);
          },
        },
        {
          ASSET_NAMESPACE: env.__STATIC_CONTENT,
          ASSET_MANIFEST: assetManifest,
          cacheControl: {
            browserTTL: ttl,
            edgeTTL: ttl,
          },
        }
      );

      return response;

      // @todo the whole logic looks strange, add test, write what it solves
      if (asset) {
        finalRequest = new Request(
          url.origin + "/" + asset,
          request as RequestInit
        );
      }

      const response = await getAssetFromKV(
        {
          request: finalRequest,
          waitUntil(promise) {
            return ctx.waitUntil(promise);
          },
        },
        {
          ASSET_NAMESPACE: env.__STATIC_CONTENT,
          ASSET_MANIFEST: assetManifest,
          cacheControl: {
            browserTTL: ttl,
            edgeTTL: ttl,
          },
        }
      );

      return response;

      // @todo the whole logic looks strange, add test, write what it solves
      if (asset) {
        finalRequest = new Request(
          url.origin + "/" + asset,
          request as RequestInit
        );
      }

      const response = await getAssetFromKV(
        {
          request: finalRequest,
          waitUntil(promise) {
            return ctx.waitUntil(promise);
          },
        },
        {
          ASSET_NAMESPACE: env.__STATIC_CONTENT,
          ASSET_MANIFEST: assetManifest,
          cacheControl: {
            browserTTL: ttl,
            edgeTTL: ttl,
          },
        }
      );

      return response;

KV is created by using

[site]
bucket = "./public"

[site]
bucket = "./public"

[site]
bucket = "./public"

[site]
bucket = "./public"

section in toml file
and exists and filled

4453161ad26e32a617c6c6d4b837c9d9/workers/kv/namespaces/989796259c334124b2e868bd9a0c3849

4453161ad26e32a617c6c6d4b837c9d9/workers/kv/namespaces/989796259c334124b2e868bd9a0c3849

4453161ad26e32a617c6c6d4b837c9d9/workers/kv/namespaces/989796259c334124b2e868bd9a0c3849

4453161ad26e32a617c6c6d4b837c9d9/workers/kv/namespaces/989796259c334124b2e868bd9a0c3849

I

Ivan Starkov•3/15/23, 5:15 PM

This library https://github.com/cloudflare/kv-asset-handler std code is used for static assets hosting

I

Ivan Starkov•3/15/23, 5:17 PM

timings are very unstable and sometimes can be very good

I

Ivan Starkov•3/15/23, 5:17 PM

But next reload

I

Ivan Starkov•3/15/23, 5:17 PM

I

Ivan Starkov•3/15/23, 5:17 PM

Font timing (r2 with cache rule) is independent

Similar Threads

Similar Threads

Similar Threads