Why key-value data storage is so expensive comparing to other resources? KV read price - $0.50/mill

Why key-value data storage is so expensive comparing to other resources?

KV read price - $0.50/million
Durable objects (that sounds a lot more complex, "permanent" and expensive) - $0.15/million
D1 database - $0.001 / million rows reads

Why everything else, where I can store some data is cheaper? Isn't kv - a cheap, temporary, cache storage?

AAnton Why key-value data storage is so expensive comparing to other resources? KV rea...

Matt Silverlock•9/20/24, 4:13 PM

KV isn't temporary - KV storage is persistent: https://developers.cloudflare.com/kv/

Cloudflare Docs

Cloudflare Workers KV | Cloudflare Workers KV

Workers KV is a data storage that allows you to store and retrieve data globally. With Workers KV, you can build dynamic and performant APIs and websites that support high read volumes with low latency.

AAnton Why key-value data storage is so expensive comparing to other resources? KV rea...

thomasgauvin•9/20/24, 4:24 PM

And to add on to Matt's response, each key-value pair in KV can handle a much higher throughput than individual rows in D1

AntonOP•9/20/24, 4:31 PM

Hmm, ok. Is KV the best tool to store and read some cache data? Like I have a complex worker function. During its work, I have to hit several 3rd party API. And I don't want to call them on every worker execution. I want to temporary cache data. On regular server I would use redis or a simple in-memory cache. What about worker? What should I use here? KV?

AAnton Hmm, ok. Is KV the best tool to store and read some cache data? Like I have a co...

thomasgauvin•9/20/24, 4:39 PM

Yes, KV is the tool for this. there's cache api https://developers.cloudflare.com/workers/runtime-apis/cache/ but that is definitely an optimization and might get evicted from cache

AAnton Hmm, ok. Is KV the best tool to store and read some cache data? Like I have a co...

JacobMGEvans•9/21/24, 10:26 PM

Definitely something you can do, I use KV to store Tailwind off the CDN (for funsies), I try weird uses of Cloudflare products in this project all the time just to see if I can and if there is other novel things you can do https://github.com/JacobMGEvans/JacobMGEvans/blob/main/src/index.ts

JacobMGEvans•9/21/24, 10:28 PM

Additionally you can use the Worker Cron to run occasionally and purge anything thats older than a certain age you give them, since you mentioned temporary

Ashish Rawat•9/23/24, 5:50 PM

I think KV is good for storing permission level data on top of D1. Like checking for permissions of a user on each api call before accessing D1 or other db.

TigersWay•9/25/24, 7:41 AM

Still receiving the message "Daily request limit exceeded for Cloudflare Workers" for a "1000 put operations" which is in fact of 226 right now!
More than a week has passed, shouldn't we have more information?

Bọc Link VN•9/25/24, 10:35 AM

Hello every one ! Have anyone know how to rename the KV Namespace?

Bọc Link VN•9/25/24, 10:35 AM

I remember that it could be renamed in the past but now it seems not to work anymore

TTigersWay Still receiving the message "Daily request limit exceeded for Cloudflare Workers...

thomasgauvin•9/25/24, 1:53 PM

Thanks for bringing it up again. This is our top priority fix at the moment. A query that is being made to alert on free quotas is misreporting quotas, so we’re fixing that. I will keep this message thread updated. Note that the query resets every day so you may not encounter this all the time

JJacobMGEvans Additionally you can use the Worker Cron to run occasionally and purge anything ...

thipperz•9/25/24, 11:59 PM

That's interesting, how do you check the current age of a KV key? I would like to set a high cacheTtl for a key while also being able to show a 'refresh' button if the current value is cached but also 3 days old for example

Tthomasgauvin Thanks for bringing it up again. This is our top priority fix at the moment. A q...

TigersWay•9/26/24, 2:13 AM

I somehow understand it needs time to fix it, but shouldn't WE have a test period for this kind of things?

Tthipperz That's interesting, how do you check the current age of a KV key? I would like t...

JacobMGEvans•9/26/24, 2:21 AM

I'd have to look at how I've done it before but my knee jerk (from my phone) reaction without looking is probably timestamp in the Metadata, maybe something like this

async function setKVValueWithTimestamp(key, value, cacheTtl) {
  const timestamp = Date.now();
  const data = JSON.stringify({ value, timestamp });
  await KV_NAMESPACE.put(key, data, { expirationTtl: cacheTtl });
}

async function setKVValueWithTimestamp(key, value, cacheTtl) {
  const timestamp = Date.now();
  const data = JSON.stringify({ value, timestamp });
  await KV_NAMESPACE.put(key, data, { expirationTtl: cacheTtl });
}

JJacobMGEvans I'd have to look at how I've done it before but my knee jerk (from my phone) rea...

thipperz•9/26/24, 3:28 AM

Ahh, sure! Thank you! That will do it

Matt Silverlock•9/26/24, 3:11 PM

In case you missed it — here's the details on the latest Workers KV performance improvements that just shipped! https://blog.cloudflare.com/faster-workers-kv/ @Workers KV

The Cloudflare Blog

We made Workers KV up to 3x faster — here’s the data

Today, we’re announcing up to 3x faster KV hot reads, with all KV operations faster by up to 20ms. And we want to pull back the curtain and show you how we did it.

AZipr•9/26/24, 3:16 PM

Wow!

Thats a massive gain!

Devon•9/26/24, 3:16 PM

that's huge! very nice!

Devon•9/26/24, 3:16 PM

Thank you!

MMatt Silverlock In case you missed it — here's the details on the latest Workers KV performance ...

thomasgauvin•9/26/24, 5:11 PM

We're super excited for this ship, and we're going to continue to make KV super fast for your data globally. The details of how we did it are all in the blog, but the graphs really show the impact

Brian Ball•9/27/24, 6:51 PM

In the kv docs /getting-started

I can't get this command to work:

npx wrangler kv key put --binding=<BINDING_NAME> "<KEY>" "<VALUE>"

just trying to follow the kv-tutorial

I used kv_binding_name - trying to follow the docs.

Though the docs say: name it BINDING_NAME - the very next step shows a create command with kv_binding_name

✘ [ERROR] A namespace with binding name "kv_binding_name" was not found in the configured "kv_namespaces".

✘ [ERROR] A namespace with binding name "kv-tutorial-kv_binding_name" was not found in the configured "kv_namespaces".

Brian Ball•9/27/24, 6:52 PM

It does work with the ID - but, I'm just reporting broken documentation continuity.

JamieD132•9/28/24, 7:01 AM

help I am trying to import kv to my javascript worker with

import { KVNameSpace } from '@cloudflare/kv-name-space'

import { KVNameSpace } from '@cloudflare/kv-name-space'

but it doesn;t work

JJamieD132 help I am trying to import kv to my javascript worker with `import { KVNameSpace...

tadhglewis•9/28/24, 8:38 AM

it doesn;t work

@JamieD132 What is the error you're getting and what do you expect to get?

tadhglewis•9/28/24, 8:39 AM

The package mentioned doesn't even exist

https://www.npmjs.com/package/@cloudflare/kv-name-space

kindaraffy•9/28/24, 3:27 PM

Hi guys! The documentation is a little murky on this one, how do I specify the expiration on the KV PUT REST body? https://developers.cloudflare.com/api/operations/workers-kv-namespace-write-key-value-pair-with-metadata

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

Kkindaraffy Hi guys! The documentation is a little murky on this one, how do I specify the e...

Marcus•9/28/24, 7:57 PM

Ooh yeah they should have an example body there, for sure. I assume it's the same as the

wrangler kv bulk put

wrangler kv bulk put

entries: https://developers.cloudflare.com/workers/wrangler/commands/#kv-bulk-put

JJamieD132 help I am trying to import kv to my javascript worker with `import { KVNameSpace...

Marcus•9/28/24, 8:02 PM

was that a suggestion from an AI tool? that's my best guess :D anyway, follow https://developers.cloudflare.com/kv/get-started/#3-bind-your-worker-to-your-kv-namespace and https://developers.cloudflare.com/kv/get-started/#5-access-your-kv-namespace-from-your-worker to use a KV namespace with a worker

Kkindaraffy Hi guys! The documentation is a little murky on this one, how do I specify the e...

thomasgauvin•9/30/24, 1:47 PM

Thanks for flagging! let me verify

thomasgauvin•9/30/24, 2:27 PM

@kindaraffy yes, expiration is supported in the REST API, we'll be fix that in docs

You can specify expiration or expirationTtl as a query param

So I can tell the users: https://api.cloudflare.com/client/v4
/accounts/{account_id}/storage/kv/namespaces/{namespace_id}/values/{key_name}?expiration=<secondsSinceEpoch>

or https://api.cloudflare.com/client/v4
/accounts/{account_id}/storage/kv/namespaces/{namespace_id}/values/{key_name}?expirationTtl=<secondsFromNow>

kindaraffy•9/30/24, 2:28 PM

Thank you.

carlos-alberto•10/1/24, 3:22 PM

Hey! Are there documented (or undocumented) rate limits on number of writes to different KV keys from a worker (on the paid plan)? We are seeing 429s when we try to write a few thousand keys within a few seconds.

thomasgauvin•10/1/24, 3:29 PM

KV has 1 write per second per key limit from the binding https://developers.cloudflare.com/kv/platform/limits/ and the APIs have global rate limits https://developers.cloudflare.com/fundamentals/api/reference/limits/#:~:text=The%20global%20rate%20limit%20for,receiving%20a%20HTTP%20429%20response.

carlos-alberto•10/1/24, 3:31 PM

Do the api rate limits apply to calls made from a worker binding?

carlos-alberto•10/1/24, 3:32 PM

We are writing to different keys, so the 1 per second should not apply

thomasgauvin•10/1/24, 3:41 PM

DM'd

ItsWendell•10/1/24, 4:41 PM

Related to the new upcoming cache purging changes, couldn't these also apply to KV?

https://blog.cloudflare.com/instant-purge/

Is it generally possible to purge a specific KV key? I think if a KV value is locally cached, and you update this it could in theory especially with the new purging system 'instantly' purge all regional KV caches right? Think this could bring quite some benefits to keep KV reads latency.

E.g. I need super quick access to a 'configuration' stored in KV, this 'configuration' is updated at most a couple of times a day. If we can keep this in the cache until this value is updated or deleted, it would speed up calls to KV a lot. Currently it's documented that KV updates do not purge KV caches, in this case it would be great if it did!

thomasgauvin•10/2/24, 7:37 PM

Thanks for the feedback @ItsWendell, like you mentioned, KV doesn't have cache purges but it's something we're keeping our eyes on

w3bcode•10/3/24, 1:03 PM

can we use https://github.com/cloudflare/cloudflare-typescript with local wrangler created kv namespace ?

GitHub

GitHub - cloudflare/cloudflare-typescript: The official Typescript ...

The official Typescript library for the Cloudflare API - cloudflare/cloudflare-typescript

Ww3bcode can we use https://github.com/cloudflare/cloudflare-typescript with local wrangl...

Hello, I’m Allie!•10/3/24, 2:30 PM

No, since it does not expose an HTTP API

w3bcode•10/3/24, 2:46 PM

ohh, got it, thanks

AAshish Rawat I think KV is good for storing permission level data on top of D1. Like checking...

thousandmiles•10/6/24, 2:47 AM

i am guessing maybe DB is more suitable for production level deployment for permissions? to ensure consistency.

Tthousandmiles i am guessing maybe DB is more suitable for production level deployment for perm...

Isaac McFadyen•10/6/24, 2:49 AM

The "inconsistency" of KV would just mean that an update might take 60 seconds to propogate globally. If you're OK with 60-second delays when adding or removing permissions then KV would still be suitable, but if not then KV isn't for you.

IIsaac McFadyen The "inconsistency" of KV would just mean that an update might take 60 seconds t...

thousandmiles•10/6/24, 2:52 AM

yes, I did assume permissions are usually crucial for production environment best with zero latency and 100% consistency.
what about D1? that said, I actually stick to D1 for auth data for my development project... simply because D1 offers more free allowances.

Isaac McFadyen•10/6/24, 2:58 AM

D1 is single-region currently, so while you'd get consistency there you're definitely not going to get zero latency.

Isaac McFadyen•10/6/24, 2:59 AM

In general there's no such thing as zero-latency, 100% consistency - all databases are just trading one for the other, some fully (i.e. KV prioritizes speed, D1 prioritizes consistency) and some partially (i.e. in etcd you can choose to read possibly-stale values or consistent values and your app can make that choice itself)

Tthousandmiles yes, I did assume permissions are usually crucial for production environment bes...

Ashish Rawat•10/6/24, 6:21 PM

Permissions doesn't change often. Firebase auth token cache auth tokens upto 1 hr by default. 60sec doesn't seems to be problem.

Isaac McFadyen•10/6/24, 6:38 PM

Yeah IMO 60 seconds is perfectly OK for permission use-cases. Even if you need to revoke permissions from a user quickly because of an issue, 60 seconds is very quick and won't give them much time to do anything dangerous.

Isaac McFadyen•10/6/24, 6:39 PM

You can also implement it as a tiered system: for most permission requests you can check KV, but then if there's something extremely sensitive (like billing details or a purchase or something) you can go back to a central DB that is consistent.

Why key-value data storage is so expensive comparing to other resources? KV read price - $0.50/mill

Similar Threads

Similar Threads

Similar Threads