what is the purpose of metadata for key value pairs cloudflare? Why would you use that verses just u

AAbakir what is the purpose of metadata for key value pairs cloudflare? Why would you us...

I

Metadata can be retrieved using a

list

list

list

list call, values cannot.

I

Isaac McFadyen•8/7/24, 5:39 PM

So if you have a couple thousand keys, and store important data in the metadata, you can get all that data with one call versus iterating through and using 1000+ calls (and being billed for 1000+ calls)

IIsaac McFadyen So if you have a couple thousand keys, and store important data in the metadata,...

A

AbakirOP•8/7/24, 5:40 PM

This makes sense thank you

J

jacob•8/7/24, 9:55 PM

Anybody who's managed to use KV with container based DI like tsyringe? I find that the kv namespace methods break once put through tsyringe. Specifically in classes using it as a dependency. I get super weird behaviour where the methods don't throw any sort of error but seemingly just stop any execution of code coming after it.

J

jacob•8/7/24, 10:27 PM

It actually appears to be unrelated to the container. I'm seeing the same behaviour simply passing it down a constructor, the kv namespace methods become unusable.

J

jacob•8/7/24, 10:45 PM

Here's an example of passing the kv down (token manager is effectively a wrapper of my particular kv namespace)

import { TokenManager, TokenType } from "@/managers/TokenManager";

export class VisionService {

  constructor(private tokenManager: TokenManager) { }

  async annotate() {
    console.log('hello world 1')
    const token = await this.tokenManager.get(TokenType.GOOGLE_CLOUD_VISION);
    console.log(token);
    console.log('hello world 2')
  }

}

import { TokenManager, TokenType } from "@/managers/TokenManager";

export class VisionService {

  constructor(private tokenManager: TokenManager) { }

  async annotate() {
    console.log('hello world 1')
    const token = await this.tokenManager.get(TokenType.GOOGLE_CLOUD_VISION);
    console.log(token);
    console.log('hello world 2')
  }

}

import { TokenManager, TokenType } from "@/managers/TokenManager";

export class VisionService {

  constructor(private tokenManager: TokenManager) { }

  async annotate() {
    console.log('hello world 1')
    const token = await this.tokenManager.get(TokenType.GOOGLE_CLOUD_VISION);
    console.log(token);
    console.log('hello world 2')
  }

}

import { TokenManager, TokenType } from "@/managers/TokenManager";

export class VisionService {

  constructor(private tokenManager: TokenManager) { }

  async annotate() {
    console.log('hello world 1')
    const token = await this.tokenManager.get(TokenType.GOOGLE_CLOUD_VISION);
    console.log(token);
    console.log('hello world 2')
  }

}

  const tokenManager = new TokenManager(c.env);

  // Logs token from kv as expected
  console.log(await tokenManager.get(TokenType.GOOGLE_CLOUD_VISION));

  const visionService = new VisionService(tokenManager);

  // Does not log token
  visionService.annotate()

  const tokenManager = new TokenManager(c.env);

  // Logs token from kv as expected
  console.log(await tokenManager.get(TokenType.GOOGLE_CLOUD_VISION));

  const visionService = new VisionService(tokenManager);

  // Does not log token
  visionService.annotate()

  const tokenManager = new TokenManager(c.env);

  // Logs token from kv as expected
  console.log(await tokenManager.get(TokenType.GOOGLE_CLOUD_VISION));

  const visionService = new VisionService(tokenManager);

  // Does not log token
  visionService.annotate()

  const tokenManager = new TokenManager(c.env);

  // Logs token from kv as expected
  console.log(await tokenManager.get(TokenType.GOOGLE_CLOUD_VISION));

  const visionService = new VisionService(tokenManager);

  // Does not log token
  visionService.annotate()

Console only outputs ..

<token>
hello world 1

<token>
hello world 1

<token>
hello world 1

<token>
hello world 1

expected

<token>
hello world 1
<token>
hello world 2

<token>
hello world 1
<token>
hello world 2

<token>
hello world 1
<token>
hello world 2

<token>
hello world 1
<token>
hello world 2

D

Darrell•8/8/24, 11:27 AM

Has expiration_ttl now gone from the V4 API (for a KV write with metadata). I can only see it in the OpenAPI spec comment and it doesn't seem to work in practice anymore.

Original message was deleted

J

jacob•8/8/24, 5:03 PM

Oh my god :NotLikeThis: thank you so much!

00xJM Were there any adjustments to how Cloudflare processed KV updates from the API l...

C

Chaika•8/9/24, 3:20 PM

https://www.cloudflarestatus.com/incidents/4lw1rrpc0fhz

Workers KV REST API Bulk Upload issues

M

mihaaai•8/13/24, 9:15 PM

Bug on the KV docs for pricing.
Since you changed the docs layout I get this misleading pricing for KV, it was 5$/million if I remember correctly, now shows

.00/million

.00/million

.00/million

.00/million
Coming from the server, it's not a front-end glitch, please fix it, or face the lawsuits

C

Chaika•8/13/24, 9:24 PM

lol thankfully the dashboard one /subscriptions still shows the right cost, should def be fixed though, thanks for reporting

Mmihaaai Bug on the KV docs for pricing. Since you changed the docs layout I get this mis...

K

kian•8/13/24, 9:38 PM

Fix is on the way, thanks for reporting it!

Mmihaaai Bug on the KV docs for pricing. Since you changed the docs layout I get this mis...

K

Kody•8/13/24, 10:11 PM

Fix is merged, should be live in ~10 min.

Thanks so much for reporting

Z

Zack•8/15/24, 9:08 PM

Can someone give some idea of what the rate limit is for writing to KV using the API? Trying to pre-populate the KV store with around 2.8k entries prior to deployment, but hitting rate limits.

Z

Zack•8/15/24, 9:08 PM

Would using the bulk API avoid this?

Z

Zack•8/15/24, 9:09 PM

I was using the individual API because I wanted to stream the body to fetch

Z

Zack•8/16/24, 10:39 AM

The above question I think I found the answer as API's appear to be limited to 1200 requests/5 minutes for basically any API on cloudflare.

Z

Zack•8/16/24, 10:41 AM

I do have another question though about the architecture of KV. I know that KV gets are cached in the local colo CDN, but I was wondering where cold requests come from. Is KV single-homed per namespace, or is the KV data distributed to multiple locations globally to reduce latency for cold hits?

Z

Zack•8/16/24, 10:41 AM

This seems to indicate that it is multi-homed where it says "Central data stores", just wondering if this is accurate and where exactly those "homes" are?

ZZack This seems to indicate that it is multi-homed where it says "Central data stores...

H

Hard@Work•8/16/24, 12:25 PM

EU and US are the two central stores

HHard@Work EU and US are the two central stores

Z

Zack•8/16/24, 12:25 PM

I see, are there any plans to expand this?

ZZack I see, are there any plans to expand this?

H

Hard@Work•8/16/24, 12:25 PM

Not that I know of?

Z

Zack•8/16/24, 12:25 PM

We have quite a lot of users from Asia so the initial hits for those areas are pretty poor

ZZack We have quite a lot of users from Asia so the initial hits for those areas are p...

T

thomasgauvin•8/16/24, 12:42 PM

Got it, thanks for the feedback Zack, we'll make sure to consider this in future plannings but it is not on the roadmap at the moment

A

Andrei Zienko•8/18/24, 2:34 PM

Guys, something eats my money, something does 20 000 reads per minute due to "Account details
>Read operations" is there any way at least understand the name of problematic KV?
ideallly with name of worker?

I

Isaac McFadyen•8/18/24, 3:14 PM

@Andrei Zienko let's move here

I

Isaac McFadyen•8/18/24, 3:15 PM

I don't think KV has stats on which worker is making those reads unfortunately

I

Isaac McFadyen•8/18/24, 3:15 PM

~~I'm not even sure it has per-namespace analytics (I could be wrong)~~ The GraphQL API has per namespace analytics, but not which Worker is making those calls

A

Andrei Zienko•8/18/24, 3:16 PM

at least on per namespace ?

Otherwise, it can be situation that , due to my statistics I have 1000 requests per day
due to your statistics it will be 1 000 000, if it is free it is not a problem, but I already pay for it..

A

Andrei Zienko•8/18/24, 3:18 PM

GraphQL API ?
I will check thanks a lot, if you have information about the right name of metrics
for me name of namespace will be enough, I will just stop workers one buy one

IIsaac McFadyen I don't think KV has stats on which worker is making those reads unfortunately

A

Andrei Zienko•8/18/24, 3:34 PM

Isaac, where to ask about the right GraphQL section query for that?
I see GetWorkersAnalytics example, but if you start search with that keyword of Cloudflare documentation website you will see only this example, where to get the right sections and parameters? GraphQL is language of queries, but where to read about them?

AAndrei Zienko Isaac, where to ask about the right GraphQL section query for that? I see GetWor...

C

Chaika•8/18/24, 5:45 PM

kvOperationsAdaptiveGroups

kvOperationsAdaptiveGroups

kvOperationsAdaptiveGroups

kvOperationsAdaptiveGroups
You can use clients like GraphiQL to explore the schema, https://developers.cloudflare.com/analytics/graphql-api/getting-started/explore-graphql-schema/, there's also a community ran site mining the schema and displaying it:
https://cfdata.lol/graphql/#AccountKvOperationsAdaptiveGroups

GraphQL - Cloudflare Datamining

Discover Cloudflare's full GraphQL Schema

C

Chaika•8/18/24, 5:47 PM

example query to see request counts since 2024-08-01 per namespace

{
  viewer {
    accounts(filter: {accountTag: "<accountId, right of workers & pages dash>"}) {
     kvOperationsAdaptiveGroups(filter: { date_gt: "2024-08-01"}, limit: 100) {
      dimensions {
        namespaceId
      }
      sum {
        requests
      }
  
      }
    }
  }
}

{
  viewer {
    accounts(filter: {accountTag: "<accountId, right of workers & pages dash>"}) {
     kvOperationsAdaptiveGroups(filter: { date_gt: "2024-08-01"}, limit: 100) {
      dimensions {
        namespaceId
      }
      sum {
        requests
      }
  
      }
    }
  }
}

{
  viewer {
    accounts(filter: {accountTag: "<accountId, right of workers & pages dash>"}) {
     kvOperationsAdaptiveGroups(filter: { date_gt: "2024-08-01"}, limit: 100) {
      dimensions {
        namespaceId
      }
      sum {
        requests
      }
  
      }
    }
  }
}

{
  viewer {
    accounts(filter: {accountTag: "<accountId, right of workers & pages dash>"}) {
     kvOperationsAdaptiveGroups(filter: { date_gt: "2024-08-01"}, limit: 100) {
      dimensions {
        namespaceId
      }
      sum {
        requests
      }
  
      }
    }
  }
}

You'd get a response like

            {
              "dimensions": {
                "namespaceId": "b2a58c72fd564bc884d506f4c6c99fed"
              },
              "sum": {
                "requests": 82018
              }
            },
            {
              "dimensions": {
                "namespaceId": "1213472aad494c65b6b3c330aaae0b14"
              },
              "sum": {
                "requests": 327
              }
            },
            {
              "dimensions": {
                "namespaceId": "c6a6c055efd741ccacee9916533e80da"
              },
              "sum": {
                "requests": 1214719
              }

            {
              "dimensions": {
                "namespaceId": "b2a58c72fd564bc884d506f4c6c99fed"
              },
              "sum": {
                "requests": 82018
              }
            },
            {
              "dimensions": {
                "namespaceId": "1213472aad494c65b6b3c330aaae0b14"
              },
              "sum": {
                "requests": 327
              }
            },
            {
              "dimensions": {
                "namespaceId": "c6a6c055efd741ccacee9916533e80da"
              },
              "sum": {
                "requests": 1214719
              }

            {
              "dimensions": {
                "namespaceId": "b2a58c72fd564bc884d506f4c6c99fed"
              },
              "sum": {
                "requests": 82018
              }
            },
            {
              "dimensions": {
                "namespaceId": "1213472aad494c65b6b3c330aaae0b14"
              },
              "sum": {
                "requests": 327
              }
            },
            {
              "dimensions": {
                "namespaceId": "c6a6c055efd741ccacee9916533e80da"
              },
              "sum": {
                "requests": 1214719
              }

            {
              "dimensions": {
                "namespaceId": "b2a58c72fd564bc884d506f4c6c99fed"
              },
              "sum": {
                "requests": 82018
              }
            },
            {
              "dimensions": {
                "namespaceId": "1213472aad494c65b6b3c330aaae0b14"
              },
              "sum": {
                "requests": 327
              }
            },
            {
              "dimensions": {
                "namespaceId": "c6a6c055efd741ccacee9916533e80da"
              },
              "sum": {
                "requests": 1214719
              }

CChaika `kvOperationsAdaptiveGroups` You can use clients like GraphiQL to explore the sc...

A

Andrei Zienko•8/19/24, 5:55 AM

Thanks, a lot
after the night without sleep I have done this by myself,
please add to documentation that JetBrains' Idea' Graphql client also works and also allow to discover schema, I've spend near 2 hours by searching the right client which will show me the schema

A

Andrei Zienko•8/19/24, 1:08 PM

Good day, yesterday I've posted my request about unusual KV read operations, I've spent this night trying to understand how to use GraphQL API, I've found problematic KV
I've fixed I think all, despite that I still see 8 million requests per 2 hours.
The situation looks crazy, is there any way to detect whi worker write a lot? Does somebody knows what to do?
My worker's statistics show less than thousand reqests per hour, not 8 millions

A

Andrei Zienko•8/19/24, 1:10 PM

Additionally cloudflare dashboard now unavaliable, but I see requests in GraphqlAnalytics API
the number of requests is so high that I am afraid I can not pay for them

A

Andrei Zienko•8/19/24, 1:20 PM

I can not write to support because dashboard does not work, is there any other way?

T

thomasgauvin•8/19/24, 1:30 PM

Hey Andrei! You've found the GraphQL API before we've officially unveiled it, we'll be adding to our docs shortly!

T

thomasgauvin•8/19/24, 1:31 PM

Regarding your ask for how to get the Worker that is making the request to the KV storage, that's not something that we have in terms of information. The recommendation here would be to add logging in your Worker, or perhaps onboard your Worker onto baselime to understand the subrequests being made

A

Andrei Zienko•8/19/24, 1:38 PM

I have all the statistics
I have 1000 requests per hour
due to cloudflare it is more than 8 000 000, I need help
I need to stop somthing, I don't know what...

A

Andrei Zienko•8/19/24, 1:38 PM

due to all my worker's statistics I don't have high traffic at all

T

thomasgauvin•8/19/24, 2:07 PM

The dashboard should be working to create a support ticket. Have you identified which KV namespace is getting the request load?

A

Andrei Zienko•8/19/24, 2:48 PM

yes, identified

A

Andrei Zienko•8/19/24, 2:48 PM

f39497e903f8434186e30c8025391826

A

Andrei Zienko•8/19/24, 2:49 PM

Can not create ticket, it does not work

AAndrei Zienko Can not create ticket, it does not work

I

Isaac McFadyen•8/19/24, 2:52 PM

Can you give the specific error it's giving you when trying to create a ticket?

A

Andrei Zienko•8/19/24, 2:55 PM

A

Andrei Zienko•8/19/24, 2:55 PM

C

Chaika•8/19/24, 2:56 PM

try different browser/clear cache? That happens on any load of the dashboard from any page?

what is the purpose of metadata for key value pairs cloudflare? Why would you use that verses just u

Similar Threads

Similar Threads

Similar Threads