How would I store an image inside KV? Not using cloudflare images

BBefus How would I store an image inside KV? Not using cloudflare images

I

Isaac McFadyen•12/20/23, 12:01 PM

You'd probably encode it as an ArrayBuffer. Keep in mind that will buffer the image into the Worker's RAM, but I don't think there's any way to stream directly to/from KV.

I

Isaac McFadyen•12/20/23, 12:01 PM

Oh actually, there is!

I

Isaac McFadyen•12/20/23, 12:01 PM

You can directly write a ReadableStream: https://developers.cloudflare.com/kv/api/write-key-value-pairs/

I

Isaac McFadyen•12/20/23, 12:02 PM

So you'd look at your request, see if it's an image that you want to upload (based on whatever factors you want), then do a

env.KV_BINDING.put("imagename", request.body)

env.KV_BINDING.put("imagename", request.body)

env.KV_BINDING.put("imagename", request.body)

env.KV_BINDING.put("imagename", request.body)

I

Isaac McFadyen•12/20/23, 12:03 PM

And then to read it back you can do a

const image = env.KV_BINDING.get("imagename", { type: "stream" });

const image = env.KV_BINDING.get("imagename", { type: "stream" });

const image = env.KV_BINDING.get("imagename", { type: "stream" });

const image = env.KV_BINDING.get("imagename", { type: "stream" }); (notice the stream part to get back as a stream)

B

BefusOP•12/20/23, 12:04 PM

Damn I'm going to try this, I will let you know!

I

Isaac McFadyen•12/20/23, 12:05 PM

Good luck!

IIsaac McFadyen Good luck!

B

BefusOP•12/20/23, 1:35 PM

export async function POST(req: NextRequest) {
  const formData = await req.formData();
  const file = formData.get("file") as File;
  console.log(file);
  console.log(file.type);
  console.log(file.name);
  // check if incoming data is a file and an image extension
  if (!file || !file.type.startsWith("image/")) {
    return new Response(JSON.stringify("error"), {
      headers: {
        "content-type": "application/json",
      },
    });
  }

  // if it is a file, upload the name of it to KV
  const { GALLERY } = process.env;
  const response = await GALLERY.put(file.name, file as unknown as ReadableStream<Uint8Array>);

  return new Response(JSON.stringify(response), {
    headers: {
      "content-type": "application/json",
    },
  });
}

export async function POST(req: NextRequest) {
  const formData = await req.formData();
  const file = formData.get("file") as File;
  console.log(file);
  console.log(file.type);
  console.log(file.name);
  // check if incoming data is a file and an image extension
  if (!file || !file.type.startsWith("image/")) {
    return new Response(JSON.stringify("error"), {
      headers: {
        "content-type": "application/json",
      },
    });
  }

  // if it is a file, upload the name of it to KV
  const { GALLERY } = process.env;
  const response = await GALLERY.put(file.name, file as unknown as ReadableStream<Uint8Array>);

  return new Response(JSON.stringify(response), {
    headers: {
      "content-type": "application/json",
    },
  });
}

export async function POST(req: NextRequest) {
  const formData = await req.formData();
  const file = formData.get("file") as File;
  console.log(file);
  console.log(file.type);
  console.log(file.name);
  // check if incoming data is a file and an image extension
  if (!file || !file.type.startsWith("image/")) {
    return new Response(JSON.stringify("error"), {
      headers: {
        "content-type": "application/json",
      },
    });
  }

  // if it is a file, upload the name of it to KV
  const { GALLERY } = process.env;
  const response = await GALLERY.put(file.name, file as unknown as ReadableStream<Uint8Array>);

  return new Response(JSON.stringify(response), {
    headers: {
      "content-type": "application/json",
    },
  });
}

export async function POST(req: NextRequest) {
  const formData = await req.formData();
  const file = formData.get("file") as File;
  console.log(file);
  console.log(file.type);
  console.log(file.name);
  // check if incoming data is a file and an image extension
  if (!file || !file.type.startsWith("image/")) {
    return new Response(JSON.stringify("error"), {
      headers: {
        "content-type": "application/json",
      },
    });
  }

  // if it is a file, upload the name of it to KV
  const { GALLERY } = process.env;
  const response = await GALLERY.put(file.name, file as unknown as ReadableStream<Uint8Array>);

  return new Response(JSON.stringify(response), {
    headers: {
      "content-type": "application/json",
    },
  });
}

I'm new to using KV, I seem to be getting the type error that GALLERY.put is not a function. But it does exist and I'm pretty sure it would work on production but I am testing locally now using npx cf-bindings-proxy

I

Isaac McFadyen•12/20/23, 1:42 PM

Hmm you're using Next?

I

Isaac McFadyen•12/20/23, 1:43 PM

You might want to ask in #next-on-pages, I haven't used Next at all and don't know how the bindings are supposed to work.

I

Isaac McFadyen•12/20/23, 1:43 PM

Also you shouldn't need to cast using Typescript - if you have the file as a File then it should be insertable by

file.readable

file.readable

file.readable

file.readable I think?

B

BefusOP•12/20/23, 1:51 PM

Yea I'm using next, I will try to ask in the other channel.

There doesn't seem to be a file.readable but there is file.stream()?

IIsaac McFadyen And then to read it back you can do a `const image = env.KV_BINDING.get("imagena...

B

BefusOP•12/20/23, 2:51 PM

How would I get all images using .list?

BBefus Yea I'm using next, I will try to ask in the other channel. There doesn't seem ...

I

Isaac McFadyen•12/20/23, 2:58 PM

That's probably it, yeah.

BBefus How would I get all images using .list?

I

Isaac McFadyen•12/20/23, 2:59 PM

list

list

only shows you the keys. You need to run

get

get

get

get on each key to get the image.

B

BefusOP•12/20/23, 3:06 PM

Happy me :)

Original message was deleted

C

Chaika•12/21/23, 12:36 AM

Web Assets, Cloudflare Pages uses KV for all uploaded assets. Lots of reads, not many writes

Original message was deleted

C

Chaika•12/21/23, 12:39 AM

I wouldn't think so. It depends on size and your budget I suppose, but R2 has a single location whereas KV has two central stores (one in NA, one in EU) and a built in caching layer.
For your own use, R2 may be cheaper though, at the cost of latency.

C

Chaika•12/21/23, 12:41 AM

There's a nice post on how KV works here:
https://developers.cloudflare.com/kv/learning/how-kv-works/
and overall storage options on CF
https://developers.cloudflare.com/workers/platform/storage-options/

C

Chaika•12/21/23, 12:42 AM

You can cache it via a custom domain, but each bucket still only lives in one region/location

C

Chaika•12/21/23, 12:44 AM

KV has some other cool things with Auto expiring keys via ExpirationTTL. You can have something be cached for x time and automatically be deleted to be repopulated. KV also as noted on that page can be used for configuration, like flags you change to activate features on your website

E

EdFletcher•12/21/23, 2:24 AM

you needn't even think of them as either/or. KV and R2 can work well together. for example say you have large binary blobs ("files") with associated structured metadata: you can store the metadata in KV with a pointer to the blob in R2. similarly with D1, you can have pointers to PKs (etc.) in a D1 DB from KV to help link K/V-style storage with relational storage.

Original message was deleted

E

EdFletcher•12/21/23, 2:51 AM

you bet, i'm glad it was helpful! and in fact that wasn't a contrived example either, i do those things in a set of workers i have open-sourced so you're welcome to take a gander at the code if it can be further helpful https://github.com/image-yinyang/

M

moody 💭•12/24/23, 2:39 PM

i could really use kv batch requests. i'm probably up to 300 requests per api call. and suspect making that many requests has a 1s+ latency overhead

Mmoody 💭i could really use kv batch requests. i'm probably up to 300 requests per api ca...

M

Matt Silverlock•12/24/23, 4:28 PM

What are you fetching that requires that many calls per eyeball/client request? Have you considered packing them into a larger value (and thus fewer keys) - ?

A batch read API won’t immediately solve performance issues as there isn’t a guarantee all of the keys in a batch would be hot (cached) or co-located in storage. Your batch would always be limited by the slowest read, identical to batch reads in most storage systems.

MMatt Silverlock What are you fetching that requires that many calls per eyeball/client request? ...

M

moody 💭•12/24/23, 4:29 PM

it would at least save cpu time

Mmoody 💭it would at least save cpu time

M

Matt Silverlock•12/24/23, 4:29 PM

Network calls don’t count towards CPU time on the Standard usage model.

M

moody 💭•12/24/23, 4:29 PM

constructing and firing 300 calls has some non-negligible overhead

M

moody 💭•12/24/23, 4:29 PM

i'd imagine

M

moody 💭•12/24/23, 4:30 PM

vs creating an array of 300 keys and firing one call

M

moody 💭•12/24/23, 4:31 PM

and i'm sure there is overhead in the backend for each call as well

Mmoody 💭constructing and firing 300 calls has some non-negligible overhead

M

Matt Silverlock•12/24/23, 4:31 PM

That is minimal vs the performance / latency issue you called out, which is why I’m suggesting you re-think your key design. Batch reads are not a panacea.

M

moody 💭•12/24/23, 4:34 PM

the use case is, i have a request that requires several megabytes of data from a postgresql database. i cache as much as i can in the worker memory, but for a fresh worker, i have to get all that data out of postgres.

i have 1 query that tells me all the keys that have new data (something like last updated timestamp), and then another query that gets the new data for all the keys that have seen updates.

my postgres database started getting bandwidth throttled (blue line) because of the workers fetching so much data from it. so i started storing the data for each key + last updated in kv and fetch from kv first

M

moody 💭•12/24/23, 4:36 PM

it is perfect for kv or redis because it's write-once (key+last-updated-ts w/ ttl)->data.

MMatt Silverlock That is minimal vs the performance / latency issue you called out, which is why ...

M

moody 💭•12/24/23, 4:39 PM

i think a batch read request that only reads from the most local region cache and then asynchronously requests copying to that datacenter would be ok

M

moody 💭•12/24/23, 5:52 PM

the other part of the latency is doing a put of each of the keys that wasnt found in the store. batch read and write are pretty important i think

MMatt Silverlock That is minimal vs the performance / latency issue you called out, which is why ...

M

moody 💭•12/26/23, 1:04 PM

curious if you have any ideas of how i can improve my architecture

M

moody 💭•12/26/23, 1:05 PM

the batch endpoint will definitely speed things up and separately i think it's also not great that multiple requests will simultaneously update the cache

MMatt Silverlock That is minimal vs the performance / latency issue you called out, which is why ...

M

moody 💭•12/26/23, 4:44 PM

i think i might try using the request response cache for the data since i don't need it replicated across regions

J

jacob•12/26/23, 10:54 PM

Is it normal for put() to take several minutes? I executed it with seemingly no errors or issues maybe 5 minutes ago or so and they still don't show up on the cloudflare dashboard

K

kian•12/26/23, 10:58 PM

How did you call put()? In a published Worker?

Kkian How did you call put()? In a published Worker?

J

jacob•12/26/23, 10:58 PM

The worker is published though this code is running locally

K

kian•12/26/23, 10:59 PM

That’s expected then, local has a local emulation of storage.

K

kian•12/26/23, 10:59 PM

If you want to run it on the edge and talk to real resources, add --remote

J

jacob•12/26/23, 10:59 PM

Oh I see, that actually sounds quite useful. Thanks!

B

BefusOP•12/27/23, 10:51 AM

I am trying to store images, should I store them directly into a KV or would it be better to store in R2 and save the link to it in a KV

BBefus I am trying to store images, should I store them directly into a KV or would it ...

A

arch•12/27/23, 11:13 AM

Probably best to save to R2 and reference them in KV. Storing large blobs in any database is generally not advised.

Aarch Probably best to save to R2 and reference them in KV. Storing large blobs in any...

I

Isaac McFadyen•12/27/23, 2:47 PM

I would normally agree with you, but KV is designed to handle blobs up to 25MiB and will be a fair bit faster than R2 if you do

(and more expensive so depends on priorities I guess)

I

Isaac McFadyen•12/27/23, 2:47 PM

That's how Pages stores all assets

IIsaac McFadyen I would normally agree with you, but KV is designed to handle blobs up to 25MiB ...

A

arch•12/27/23, 3:28 PM

True! In my head I always avoid putting files in any database/kv store because then my code has to deal with retrieving, potentially decoding and then serving it up instead of offloading it to the/a static file CDN (although you might still want a worker for generating a signed S3 URL).

How would I store an image inside KV? Not using cloudflare images

Similar Threads

Similar Threads

Similar Threads