Hi, we are using KV for caching image blobs. Once in a while we experience that the data we stored i

Hi, we are using KV for caching image blobs. Once in a while we experience that the data we stored in KV is corrupt. We are trying to figure out if that happens before or during saving the data in KV - Unlike R2 we cannot provide a checksum when writing to KV so we are wondering if KV does some sort of integrity check under the hood? Is it possible that data could be written “half way” and get corrupted if the writing worker crashes or similar?

TTilon Hi, we are using KV for caching image blobs. Once in a while we experience that ...

Matt Silverlock•2/1/24, 2:21 PM

This would be surprising to us (we have never had any reports) - how are you assessing corruption? In other words when you retrieve a key value pair, how do you know it’s corrupted?

Mmatt_d Yes, the `prefix` argument to `list()`. (Of course this requires knowing about `...

Darick•2/1/24, 10:09 PM

Thank you MattD.

Within a prefix, though, I want to be able to specify start (and end) points similar to what the Durable Object Transactional Storage API offers: https://developers.cloudflare.com/durable-objects/api/transactional-storage-api/#supported-options-2

Is this possible in KV, or is it something that might be added in the future?

Transactional Storage · Cloudflare Durable Objects docs

The Transactional Storage API allows you to achieve consistent key-value storage.

bz_008•2/6/24, 8:20 AM

how to get the first and last keys from a KV?

Bbz_008 how to get the first and last keys from a KV?

Hello, I’m Allie!•2/6/24, 8:20 AM

You would have to list through the namespace

bz_008•2/6/24, 8:22 AM

I know, but I still don't know how to get first and last one since [0] and [length-1] don't work

Bbz_008 I know, but I still don't know how to get first and last one since [0] and [leng...

Hello, I’m Allie!•2/6/24, 8:23 AM

You have to run list multiple times if you have more than 1k keys

bz_008•2/6/24, 8:23 AM

yeah, that's another issue

bz_008•2/6/24, 8:25 AM

KV looks a little hard to use to me..

Bbz_008 yeah, that's another issue

Hello, I’m Allie!•2/6/24, 8:27 AM

Like this:

async function getFirstAndLast(KV: KVNamespace) {
    let listResult = await KV.list();
    const first = listResult.keys[0];
    while(!listResult.list_complete) {
        listResult = await KV.list({cursor: listResult.cursor});
    }
    const last = listResult.keys[listResult.keys.length - 1];
    return { first, last };
}

async function getFirstAndLast(KV: KVNamespace) {
    let listResult = await KV.list();
    const first = listResult.keys[0];
    while(!listResult.list_complete) {
        listResult = await KV.list({cursor: listResult.cursor});
    }
    const last = listResult.keys[listResult.keys.length - 1];
    return { first, last };
}

HHello, I’m Allie!Like this: ```ts async function getFirstAndLast(KV: KVNamespace) { let listR...

bz_008•2/6/24, 8:35 AM

Cool, thanks for the help

Santosh•2/8/24, 5:03 PM

Guys, I have stored a large object in KV. Definitely not 25MB limit being reached, and write it using Json.stringify. When I read it back, ocassionally Json.parse throws Unterminated string error. How do I handle this. Seems like this happens with large valiues.

Santosh•2/8/24, 5:03 PM

It does appear like JSON is not the right format for KV for large objects. What do I do?

Santosh•2/8/24, 5:05 PM

Is Json the only format to write to KV? Can someone throw some light here? This is currently happening locally, not sure if we will see this on CF>

zegevlier•2/8/24, 5:05 PM

Any reason you're not just putting the object directly into KV, instead of converting it to JSON first?

Santosh•2/8/24, 5:05 PM

Thats what the current documentation shows.

Santosh•2/8/24, 5:06 PM

Is it possible to write raw object type to KV?

zegevlier•2/8/24, 5:06 PM

I thought it was but looking at the docs I might be misremembering

zegevlier•2/8/24, 5:06 PM

let me check

Zzegevlier Any reason you're not just putting the object directly into KV, instead of conve...

Hello, I’m Allie!•2/8/24, 5:06 PM

KV only takes strings/ArrayBuffers no?

Hello, I’m Allie!•2/8/24, 5:07 PM

ReadableStreams that are stored as an ArrayBuffer

zegevlier•2/8/24, 5:09 PM

Yea I was misremembering, I thought both DOs and KV supported that but it seems only DOs do

Zzegevlier Yea I was misremembering, I thought both DOs and KV supported that but it seems ...

Hello, I’m Allie!•2/8/24, 5:10 PM

Yeah. DOs were the first to use v8.serialize. This also means that KV doesn't support any of the more complex objects

SSantosh Is Json the only format to write to KV? Can someone throw some light here? This ...

zegevlier•2/8/24, 5:10 PM

You can write to KV using any format you want, you just need to serialize it first. JSON should work though, unless you go over the 25mb limit

zegevlier•2/8/24, 5:11 PM

Are you sure the JSON stringified object also does not hit that limit?

Santosh•2/8/24, 5:21 PM

yes, its like 2 MB when I pull the data from another server prior to keeping it in KV

Santosh•2/8/24, 5:21 PM

I used postman to veriify the graphql response

Santosh•2/8/24, 5:22 PM

Can I get some alternatives to Json that works on Cloudflare worker?

Ttoinbis Hey, where can I read about KV scaling strategies? My values are simple - object...

Hello, I’m Allie!•2/10/24, 5:17 AM

I would guess that the falloff happens in the billions/trillions of keys, if ever

Hello, I’m Allie!•2/10/24, 5:17 AM

Since Pages runs on KV

Hello, I’m Allie!•2/10/24, 5:18 AM

I don’t think performance is particularly tied to the number of keys

Ttoinbis Thanks @HardlyWorkin' ! What is the performance tied to then?:)

Hello, I’m Allie!•2/10/24, 5:21 AM

Latency to the core datacenters mostly, when the key you are asking for isn’t cached

Ttoinbis Interesting! I wonder -is this aspect of KV documented anywhere?

Hello, I’m Allie!•2/10/24, 5:23 AM

That it is highly scalable?

Hello, I’m Allie!•2/10/24, 5:23 AM

Or…

Ttoinbis I mean, which are the core data centers, how do I know which values are cached a...

Hello, I’m Allie!•2/10/24, 5:30 AM

There are two cores, one in North America, one in Europe. Don’t think the actual locations are public

Hello, I’m Allie!•2/10/24, 5:31 AM

And afaik you can’t really tell if a value is stale, other than that it isn’t updated for 60+ seconds

Alvee•2/10/24, 6:15 AM

for my multitenant saas I want to store usage for each tenant usage of workers requests having only 1 worker main backend api. want to track the uages of teach tenant using their StoreId (ecommerce saas). Saving request count on each request (using hono js btw) to KV is a good idea or should I store them in a mysql or is this complete madness any better solutions ?

Alvee•2/10/24, 6:38 AM

how does cloudflare store workers request count. :blob_sweat:

AAlvee how does cloudflare store workers request count. :blob_sweat:

Hello, I’m Allie!•2/10/24, 6:46 AM

I would guess it is stored in CF's internal version of AnalyticsEngine

AAlvee for my multitenant saas I want to store usage for each tenant usage of workers r...

Hello, I’m Allie!•2/10/24, 6:46 AM

I would probably go with a Durable Object per customer. With KV, you aren't guaranteed atomic writes, so increments could be overwritten

HHello, I’m Allie!I would probably go with a Durable Object per customer. With KV, you aren't guar...

Alvee•2/10/24, 7:14 AM

another concern of mine is people dos attack from localhost or unknown origins. is it possible to only allow traffic from tenant website domain in workers. IG maybe I have to set this up in my api domain security WAF /?

AAlvee another concern of mine is people dos attack from localhost or unknown origins. ...

Hello, I’m Allie!•2/10/24, 7:16 AM

You can check the origin header, but that does not prevent someone from spoofing said header. Not too difficult to do if you are using curl

HHello, I’m Allie!You can check the origin header, but that does not prevent someone from spoofing...

Alvee•2/10/24, 7:18 AM

I mean checking inside worker mean request already made. no point in there

Hi, we are using KV for caching image blobs. Once in a while we experience that the data we stored i

Similar Threads

Similar Threads

Similar Threads