Any chance to consider this feature to Filter/search KV by metadata tags - https://github.com/clou

Any chance to consider this feature to Filter/search KV by metadata tags -

https://github.com/cloudflare/workers-sdk/issues/3118

GitHub

🚀 Feature Request: Filter/search KV by metadata tags · Issue #3118 ...

Describe the solution Describe the solution To be able to use Wrangler CLI in GitHub actions and target specific keys in KV by their metadata(ideally to be able to delete them based on a metadata v...

VVikash Any chance to consider this feature to Filter/search KV by metadata tags - htt...

VikashOP•11/24/23, 4:45 AM

I was working on small URL shortner project and without metadata search/filter, I'd need to use D1 to separately manage which key belong to what customer to display the list on dashboard.

sten•11/25/23, 1:54 AM

Are there any blog posts where people go into thinks to think about when designing good keys for efficient KV usage?

sten•11/25/23, 2:05 AM

This is a very good question btw
https://community.cloudflare.com/t/fetching-a-kv-value-using-a-wildcard-key-or-pattern/371495

manishrc•11/25/23, 5:43 AM

Re: High latency for KV

I'm experiencing a latency of 20ms-200ms for KV.get, even when there's only one record and multiple requests are made from the same location. The value remains unchanged throughout. The lowest latency I've observed is 14ms, but it's infrequent.

Here's how I measured:

export default {
  async fetch(request, env, ctx) {
    const time_start = Date.now();
    const data = await env.KV.get("test");
    const latency = time_start - Date.now();
    return new Response(JSON.stringify({ data, latency }));
  },
};

export default {
  async fetch(request, env, ctx) {
    const time_start = Date.now();
    const data = await env.KV.get("test");
    const latency = time_start - Date.now();
    return new Response(JSON.stringify({ data, latency }));
  },
};

The latency response should remain unaffected by any delays in the request. Given the current response rate, it would be more efficient to utilize the database.

What mistake am I making?

Mmanishrc **❓ Re: High latency for KV** I'm experiencing a latency of 20ms-200ms for KV.g...

SK•11/25/23, 5:55 AM

there is no mistake, kv is in-fact not negligible latency like you might assume

SK•11/25/23, 5:56 AM

you can uses worker cache api to cache this for much better latencies

SK•11/25/23, 5:57 AM

I think kv v2 may have solved this but it was pulled back, I'm waiting for it to come back so I can test it!

SSK you can uses worker cache api to cache this for much better latencies

Hello, I’m Allie!•11/25/23, 9:16 AM

Well, not really. After the first request, KV is already cached with the cache API. If you know that a record won’t change for a long time, you can set a cacheTtl

HHello, I’m Allie!Well, not really. After the first request, KV is already cached with the cache A...

SK•11/25/23, 9:17 AM

really? when I tried it last time it wasn't honored

SK•11/25/23, 9:17 AM

maybe it's because I'm in a "bad" region (Australia, so hitting the Sydney data center)

SSK really? when I tried it last time it wasn't honored

Hello, I’m Allie!•11/25/23, 9:17 AM

Note that a cacheTtl is a maximum value, as it can be evicted before then

SK•11/25/23, 9:18 AM

right yeah

SK•11/25/23, 9:18 AM

I didn't do a proper benchmark, I did a crude experiment where I logged the latencies in the response headers and just refreshed the page a bunch of times and while the latency dropped from the cold read of around 700ms it was still averaging around 20-50ms

SK•11/25/23, 9:19 AM

it jumped around a lot for me

SK•11/25/23, 9:19 AM

again, might just be my experience on a bad day with a bad region

SK•11/25/23, 9:20 AM

this matches what upstash found in their much more scientific benchmark https://upstash.com/blog/edgecaching-benchmark

HHermes Alves Hi there, Is there any nodejs SDK to consume data from Cloudflare KV? I want to ...

Hermes Alves•11/26/23, 5:32 AM

Do you query through the rest api endpoint, since it doesn't have an SDK?

VVikash Any chance to consider this feature to Filter/search KV by metadata tags - htt...

VikashOP•11/27/23, 1:03 PM

@HardlyHolidayin’ Any chance?

VVikash @HardlyHolidayin’ Any chance?

Hello, I’m Allie!•11/27/23, 1:04 PM

I don't work for Cloudflare, so I can't comment. Also, pinging people won't get you an answer any faster

VVikash Any chance to consider this feature to Filter/search KV by metadata tags - htt...

DaniFoldi•11/27/23, 5:19 PM

Without knowledge of too many internal details, I highly doubt that multi-dimension indexing is something that KV would support in the foreseeable future, it's meant to be a key-value store. If you need more advanced queries, #durable-objects or #d1-beta might be more suitable

captain•11/27/23, 9:36 PM

was thre any KV hickups today?

Unsmart•11/27/23, 9:44 PM

everything seems fine for me

UUnsmart everything seems fine for me

captain•11/27/23, 9:55 PM

its fine for me now too, but around 8:30 to 9:30 when I was using something i had built it kept giving me old data,maybe it was a hickup somewhere else idk

captain•11/27/23, 9:56 PM

i have a 3rd party site with embedded google tag which then references some css data in a worker I created

captain•11/27/23, 9:56 PM

and that data is stored in KV

captain•11/27/23, 9:56 PM

but its working now...so moving on

DDaniFoldi Without knowledge of too many internal details, I highly doubt that multi-dimens...

VikashOP•11/28/23, 8:10 AM

thanks for confirming.

Mmono_cron Hey! I have a few questions regarding the data returned from https://developers....

Matt Silverlock•11/28/23, 1:50 PM

Let me take a look.

sten•11/28/23, 5:35 PM

const key = "userId123;date;2023-01-01;postId;456";

const key = "userId123;date;2023-01-01;postId;456";

I guess this is not a good key structure for posts?
I am thinking of when I am suppose to delete a post and I only have the postId, then I have to iterate the whole DB to find a key that matches this specific key by splitting each key and looking on index[2] .

Lilitu_Kali•12/3/23, 4:33 PM

Hey all, not sure if this is a "me" issue, so checking in - has anyone else been experiencing 500 errors related to kv storage today? A worker route was working then after manually adding maybe a dozen keys max via the dashboard I started getting 500s. Not sure if they were related or it was just a coincidence. The worker basically reads through the list of keys then for each key reads it value and outputs a json formatted list of the content.

Lilitu_Kali•12/3/23, 4:34 PM

The path where the Worker is triggered is https://kol.linzinha.is/players

Lilitu_Kali•12/3/23, 5:19 PM

so weird, I was seing a different error which doesn't seem to matter but I copied and pasted the exact same code in and resaved the worker, and the 500s went away

Aries•12/4/23, 3:37 AM

https://files.aries.fyi/2023/12/03/Code_5be27562613e7080.png

Aries•12/4/23, 3:38 AM

for (let index = 0; index < 10; index++) {
  await new Promise((resolve) => setTimeout(resolve, 1000));
  await getKey(versionKey);
}

for (let index = 0; index < 10; index++) {
  await new Promise((resolve) => setTimeout(resolve, 1000));
  await getKey(versionKey);
}

Aries•12/4/23, 3:38 AM

how does that make any sense?

Aries•12/4/23, 3:41 AM

ah, next.js decided to automatically cache it.

Aries•12/4/23, 3:41 AM

thanks next.js

Aries•12/4/23, 3:42 AM

wait hold on.

Aries•12/4/23, 3:42 AM

still an issue?
https://files.aries.fyi/2023/12/03/Code_9dd93c6dc0d365c3.png

kian•12/4/23, 4:10 AM

KV is eventually consistent, and caches per-colo. This looks like normal behaviour.

Varna•12/5/23, 3:44 PM

Does data get removed from KV if not accessed for a long time? i.e. 1 year.

VVarna Does data get removed from KV if not accessed for a long time? i.e. 1 year.

Hello, I’m Allie!•12/5/23, 3:45 PM

No. As long as you are willing to pay, it will stay

Chaika•12/5/23, 3:46 PM

(Unless you set an Expiration TTL on it, of course)

Chaika•12/5/23, 3:46 PM

If you wanted to, it could. Otherwise it's a full data store. After a year it wouldn't be cached anywhere but it would still be at the two central stores

w•12/5/23, 10:53 PM

Would Cloudflare KV be secure enough to store encryption keys? I'm trying to store many sets of data with each one being encrypted so thinking of using AWS KMS to store a master encryption key and storing multiple Data Encryption Keys in Cloudflare KV that are encrypted and decrypted using the AWS KMS key before use each time. Since I need to encrypt and decrypt frequently, KMS is a bit too price prohibitive for use entirely.

Santosh•12/6/23, 3:55 PM

Quick question: In case I update a KV value behind the scenes prior to its expiration (say using a DO), would the clients who are using the KV see it upon the next fetch or is the expectation that clients using the KV see the updated value within max of 60 seconds??

export default { async fetch(request, env, ctx) { const time_start = Date.now(); const data = await env.KV.get("test"); const latency = time_start - Date.now(); return new Response(JSON.stringify({ data, latency })); }, };

Any chance to consider this feature to Filter/search KV by metadata tags - https://github.com/clou

Similar Threads