Looking at key-value benchmarks right now and found it surprising to see Cloudflare's KV latencies b

Looking at key-value benchmarks right now and found it surprising to see Cloudflare's KV latencies being so high. Is there something that these benchmarks overlook?
https://upstash.com/blog/edgecaching-benchmark
https://deno.com/blog/comparing-deno-kv

MMatt Silverlock If you have _single_ files that are larger than 25MB, you should store them in R...

ramesh prasad•1/13/24, 7:48 PM

thanks brother

ramesh prasad•1/13/24, 7:50 PM

And yeah I have an usecase to cover. After every 1000 worker invocation I need to hit an api to external system. how can i do that? Any idea on this? Workers will as usual get data from user, the data will be stored on lets say d1. And after 1000 data I get from user, I need to push that data to external system. Any idea here?

X•1/13/24, 7:52 PM

please manage every excitement. help me 1st.

Aatgctg Looking at key-value benchmarks right now and found it surprising to see Cloudfl...

Unsmart•1/13/24, 8:10 PM

I would say KV is by no means the fastest database in the world but I would take these comparisons with a grain of salt as both those companies have a vested interest in making their db looking better.

I would test KV latency for yourself the latency will depend how close you are to the origin and if the value is cached or not. For me a cache miss on KV takes roughly ~300ms and a cache hit takes roughly ~40ms.

UUnsmart I would say KV is by no means the fastest database in the world but I would take...

atgctgOP•1/13/24, 10:04 PM

thanks for the reply!

Unsmart•1/15/24, 5:48 PM

dear lord that code is very cursed KEKW

Unsmart•1/15/24, 5:55 PM

I'm very confused how they got a p99 of 700ms for doing a single query and get on DynamoDB also....

My work has serveral routes at p99 of 500ms or less that do at least 2-3x the amount of queries lol

Unsmart•1/15/24, 5:57 PM

DynamoDB is an interesting DB though where more read load produces lower latency lol so 500ms when barely any reads and 200ms p99 at peak

Unsmart•1/15/24, 6:04 PM

Also interesting to note that the Deno comparison is doing a list followed by a get for every service but then their own only does a list so very unfair comparison

Unsmart•1/15/24, 9:39 PM

I dont think theres much reason to benchmark them just pick the one you want to use. If you already use cloudflare and the feature set of KV is good enough for your use case I dont see any reason to use something outside of cloudflare. Picking based off of benchmarks is only really relevant when you are cloudflare scale imo

Unsmart•1/15/24, 9:44 PM

The p99 latency I measured was for DynamoDB not KV and I just looked at the AWS Lambda latency provided by cloudwatch.

Unsmart•1/15/24, 9:46 PM

The somewhat equivalent on cloudflare would be going to your worker and looking at execution duration but I'd imagine you probably have a lot of routes on a single worker. And the graph also shows in GB-sec instead of milliseconds.

Julian•1/17/24, 12:08 AM

One of my collegues made an SDK for Cloudflare KV API in Java! https://github.com/segu23/cloudflare-sdk-kv

GitHub

GitHub - segu23/cloudflare-sdk-kv

Contribute to segu23/cloudflare-sdk-kv development by creating an account on GitHub.

mackenzie•1/18/24, 8:31 AM

#real noob# question. I have a worker/DO that writes some status to KV. I'd like to write a small we app (and later mobile app) to visualize it....are there any examples for how to go about doing that, using cloud flare or other (using rust if possible)

mackenzie•1/18/24, 8:32 AM

Possible with pages?

mackenzie•1/18/24, 8:33 AM

(later I might want to subscribe to a Pub/Sub topic, if I get Beta access)

kp1197•1/20/24, 11:54 PM

Hi, I am setting up local development, and I have a worker which has a KV binding. Is it possible for me to develop locally, while connecting to and accessing the values for KV namespace as stored in my cloudflare account? i.e.; npx wrangler login then npx wranger dev --remote

kp1197•1/20/24, 11:54 PM

My overall goal is to follow principle of least access and not have any copies of the values in my KV namespace stored in any way in my local, even in a command buffer

kian•1/20/24, 11:57 PM

Yes, that's what remote development is

kp1197•1/21/24, 12:07 AM

Hmmm... any idea what this error message is about?

kp1197•1/21/24, 12:07 AM

notes: [
{
text: 'Uncaught ReferenceError: CONFIG is not defined\n' +
' at index.js:28:26\n' +
' [code: 10021]'
}
],
location: undefined,
kind: 'error',
code: 10021
}

kp1197•1/21/24, 12:08 AM

Preceeded by: 'A request to the Cloudflare API' ... 'failed'

kp1197•1/21/24, 12:09 AM

My toml looks like this: kv_namespaces = [
{ binding = "CONFIG", id = "<I_AINT_TELLIN>" }
]

Cyb3r-Jak3•1/21/24, 12:24 AM

What does your code look like?

kp1197•1/21/24, 12:36 AM

It has a line which references CONFIG in the top level scope

kp1197•1/21/24, 12:36 AM

I.e.; global

kp1197•1/21/24, 12:40 AM

Something like const secret = CONFIG.get("secret")

Cyb3r-Jak3•1/21/24, 12:46 AM

the variables aren't defined at global scope level, only on each request. See the example at https://developers.cloudflare.com/kv/api/read-key-value-pairs/

kp1197•1/21/24, 1:12 AM

Got it, Ill try that

CCyb3r-Jak3 the variables aren't defined at global scope level, only on each request. See th...

kp1197•1/22/24, 8:32 PM

That worked, thank you. I had rapidly coded it up, made a note to myself to double check the kv access code and then forgot to circle back and reference the docs. Appreciate the help. That was my amaetur hour moment!

kp1197•1/22/24, 10:38 PM

I'm looking a kv_namespaces for workers, and from what i understand, preview_id is an extra measure to prevent writing / reading from prod kv values. So, even if I have separate environments in my wrangler.toml for env.dev and env.production, the preview

kp1197•1/22/24, 10:39 PM

_id adds an additional layer of safety, making it impossible to write to prod kv_namespaces

kp1197•1/22/24, 10:39 PM

So here is my question:

kp1197•1/22/24, 10:40 PM

I already have a separate KV_namespace (three different IDs) for each of my environments, env.dev, env.staging and env.production. So, should I create a preview_id and assign that same preview_id to each of my kv_namespaces' preview_id? That way, no matter what environment I specify when i run npx wranger dev, it will always hit the namespace indicated by that preview_id?

kp1197•1/22/24, 10:47 PM

Creating three spearate preview_ids for dev, staging, and production seems like it would unnecessarily clutter things up

fasm•1/24/24, 12:17 PM

Hi all, I'm running the request in the KV API docs here (https://developers.cloudflare.com/api/operations/workers-kv-namespace-write-key-value-pair-with-metadata) , but I'm getting the following response:

{
  result: null,
  success: false,
  errors: [
    {
      code: 10047,
      message: "can not parse value and metadata from multipart request body: 'error getting part of multipart request, after reading 0 parts: 'multipart: NextPart: EOF''"
    }
  ],
  messages: []
}

{
  result: null,
  success: false,
  errors: [
    {
      code: 10047,
      message: "can not parse value and metadata from multipart request body: 'error getting part of multipart request, after reading 0 parts: 'multipart: NextPart: EOF''"
    }
  ],
  messages: []
}

Any ideas?

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

kian•1/24/24, 12:20 PM

What does your code/command look like?

Darick•1/24/24, 6:47 PM

Is there a way to specify a starting point when listing KV values with

list()

list()

, similar to the

start

start

startAfter

startAfter

options for Durable Object storage: https://developers.cloudflare.com/durable-objects/api/transactional-storage-api/#supported-options-2 ?

(I see the

cursor

cursor

but it's not perfect because it would require first listing all of the previous values and getting the

limit

limit

exactly right ...)

Transactional Storage · Cloudflare Durable Objects docs

The Transactional Storage API allows you to achieve consistent key-value storage.

Behunin•1/24/24, 7:49 PM

Is this the way to get permissions for all the

Workers KV Namespace

Workers KV Namespace

API endpoints or do I need an API Key?

Unsmart•1/26/24, 6:07 PM

It doesnt write to the same key so it doesnt have any delay as a new key will be able to read instantly

Mmatt_d Hi Behunin. I just tried this out on my personal account to double check. The an...

Behunin•1/26/24, 9:08 PM

Thank you for the validation.

spaone•1/26/24, 10:24 PM

I am trying to use a worker to store data with kv. I am trying to import KVNamespace from 'storage' At the beginning of the code to then use KVNamespace.create later on. However, I am getting an error from storage.

kian•1/26/24, 10:44 PM

Neither of those are a thing in Workers

kian•1/26/24, 10:44 PM

Are you referring to a specific wrapper library?

Tilon•1/31/24, 9:56 PM

Hi, we are using KV for caching image blobs. Once in a while we experience that the data we stored in KV is corrupt. We are trying to figure out if that happens before or during saving the data in KV - Unlike R2 we cannot provide a checksum when writing to KV so we are wondering if KV does some sort of integrity check under the hood? Is it possible that data could be written “half way” and get corrupted if the writing worker crashes or similar?

{ result: null, success: false, errors: [ { code: 10047, message: "can not parse value and metadata from multipart request body: 'error getting part of multipart request, after reading 0 parts: 'multipart: NextPart: EOF''" } ], messages: [] }

Looking at key-value benchmarks right now and found it surprising to see Cloudflare's KV latencies b

Similar Threads

Similar Threads

Similar Threads