I'm building something like google sheets. Each user could have multiple sheets and I could have tho

I'm building something like google sheets. Each user could have multiple sheets and I could have thousands of users. I'm thinking how to store individual rows. I would use d1, but I'm scared to hit the 10GB limit and multiple databases seems like a lot of work.

I first though to store it as kv, key being the spreadsheet_id and value being the csv. This seems fairly easy, but my issue is that with every update I need to replace the entire csv, having "eventually" consistent data will most likely lead to loosing data when multiple updates occur.

cosbgnOP•6/29/24, 3:29 PM

My final solution is:

KV, with key being spreadsheet_id/row_id and value being the csv only for that row

cosbgnOP•6/29/24, 3:30 PM

this seems fine, but it means I'll have all data of all spreadsheets in the same kv

cosbgnOP•6/29/24, 3:30 PM

is this feasable? will it become super slow?

cosbgnOP•6/29/24, 3:31 PM

I've read about prefix, are these like indexes and will they speed up my queries? so if I prefix by spreadsheet_id it will look only there and avoid looking on millions of rows?

cosbgnOP•6/29/24, 3:31 PM

any suggestions on how to do this? How would you do it?

Ccosbgn I'm building something like google sheets. Each user could have multiple sheets ...

Matt Silverlock•6/29/24, 4:21 PM

You almost definitely want strong consistency (serializable) - Durable Objects would be the fit here.

Each DO is a sheet. Use y.js to manage state/changes/apply deltas. A framework like PartyKit can help simplify this: https://www.partykit.io/

KV is almost definitely not a fit as any changes you make won’t be immediately reflected to other users.

cosbgnOP•6/29/24, 4:26 PM

Ok I'll have a look at do. Thanks!

Chiquinho2001•6/30/24, 11:02 AM

Hi everyone,

I'm currently facing issues with using KV Storage in my project. Specifically, I'm having trouble getting an image to load when I test my worker locally. I've followed the documentation, but I still can't seem to get it to work. Has anyone encountered a similar problem or have any suggestions on how to resolve this?

Thanks in advance for your help!

EEisenbergEffect When getting a value from a KV store, is there a network hop involved in that op...

Hello, I’m Allie!•6/30/24, 5:08 PM

There may be a network hop, depending on whether or not the value is cached in the local datacenter

CChiquinho2001 Hi everyone, I'm currently facing issues with using KV Storage in my project. S...

Tilon•6/30/24, 6:02 PM

When running locally in dev mode, then your local KV values are different from the remote KV running in the Cloud I think. Maybe that explains the behavior you are experiencing.

thomasgauvin•6/30/24, 6:19 PM

What type of application are you trying to build? Are you frequently accessing the same kv value? And what do your data patterns look like, how often are you writing/deleting/reading?

thomasgauvin•6/30/24, 6:56 PM

Seems like the right tool (frequent reads, infrequent writes), what are the KVs storing for you use case (feel free to dm me if you prefer a private communication channel)? How quickly after a write do you need to access the KV pair? Any updates?

TTilon When running locally in dev mode, then your local KV values are different from t...

Chiquinho2001•6/30/24, 7:57 PM

But the values of the KV storage when I test locally should not be the same as those when I test with the cloud values? The image represents the result when I run the program.

CChiquinho2001 But the values of the KV storage when I test locally should not be the same as t...

thomasgauvin•6/30/24, 8:59 PM

When you use 'wrangler dev', there's a locally emulated KV store that is used (not the cloud values), here's the docs section that explains it in detail https://developers.cloudflare.com/kv/get-started/#4-develop-locally-with-wrangler

Any cloud values will not be in your locally emulated KV. You can either populate your KV locally or run your workers in the cloud use wrangler dev --remote

Cloudflare Docs

Get started · Cloudflare Workers KV

Workers KV provides low-latency, high-throughput global storage to your Cloudflare Workers applications. Workers KV is ideal for storing user …

Tthomasgauvin When you use 'wrangler dev', there's a locally emulated KV store that is used (n...

Chiquinho2001•6/30/24, 10:15 PM

After following the instructions from the link you sent, I still haven't managed to solve the problem. I'm not sure if the image needs to be saved as a JPEG or if it has to be saved in base64 format.

thomasgauvin•6/30/24, 11:53 PM

I'm not sure, would depend on how you're saving it/using it. It does seem like it's present in the dashboard so I wonder why it's not returning with 'wrangler kv key list ...'. are you logged in to the right user with wrangler?

Chiquinho2001•7/1/24, 8:38 AM

Yes, I have the data well stored in the KV storage. I have the account login and I put everything correctly in the TOML file, but it still gives an error.

mattp•7/1/24, 9:08 AM

https://developers.cloudflare.com/fundamentals/api/reference/limits/ <- Rate limits of REST API.
Does this include worker binding calls?

CChiquinho2001 After following the instructions from the link you sent, I still haven't managed...

Matt Silverlock•7/1/24, 2:01 PM

Are you setting the correct content-typecontent-type header in the response?

MMatt Silverlock Are you setting the correct `content-type` header in the response?

Chiquinho2001•7/1/24, 2:02 PM

Thanks everyone, it's working.

Aditya Mathur•7/1/24, 6:35 PM

Hello! I am using WorkerKVWorkerKV for some experiment and this is my code

await this.namespace.put(keyWithPrefix, JSON.stringify(payload), {
  expiration: Math.floor(payload.resetTime.getTime() / 1000),
});

await this.namespace.put(keyWithPrefix, JSON.stringify(payload), {
  expiration: Math.floor(payload.resetTime.getTime() / 1000),
});

So here are the problems I am facing -

1. The key is not getting deleted, I am still getting the old value when I fetch it. Even if I just wait for it the expire without sending anymore request, it's still not getting delete.
2. If I want to update the value for a key but still want to have the same expiration time, what will be implmentation for that? Should I just pass the expirationexpiration props again?

Chaitanya•7/2/24, 1:15 PM

Hello, I would like to trigger a worker before stream is getting connected in cloudflare (rtmp ingest protocol)? Please help me is this possible ?

Iim bomu Hi ! New here, do i need workers to access KV ? Are workers script only uploaded...

Hard@Work•7/4/24, 9:49 AM

Using the cloudflarecloudflare package is probably the easiest way, though note that using it is not ideal for performance reasons, so we recommend running KV operations through a Worker

npm

cloudflare

The official TypeScript library for the Cloudflare API. Latest version: 3.4.0, last published: 15 days ago. Start using cloudflare in your project by running npm i cloudflarenpm i cloudflare. There are 81 other projects in the npm registry using cloudflare.

sadsciencee•7/5/24, 8:07 AM

hi, has anyone been able to successfully connect to remote KV from local pages dev? I'm using the --local=false flag but it's still connecting to a local KV, which doesnt work bc I am testing with a few other bindings that reference the KV in the cloud (need this to auth into shopify's local admin dev)

Ssadsciencee hi, has anyone been able to successfully connect to *remote* KV from local pages...

Cyb3r-Jak3•7/5/24, 1:52 PM

Pages doesn't support remote dev

CCyb3r-Jak3 Pages doesn't support remote dev

sadsciencee•7/5/24, 10:09 PM

oof, even for things like DO/Queues?

Ssadsciencee oof, even for things like DO/Queues?

Hello, I’m Allie!•7/5/24, 10:14 PM

That is correct, though Queues doesn’t support remote dev for Workers either

Original message was deleted

Matt Silverlock•7/7/24, 9:13 AM

Are you positive you are writing to the account ID you intend to?

Original message was deleted

Matt Silverlock•7/7/24, 10:24 AM

Emit the constructed URL and the serialized body you’re sending.

Matt Silverlock•7/7/24, 10:25 AM

If it’s not showing in the dash for that account… it’s not being written.

Original message was deleted

Matt Silverlock•7/7/24, 10:55 AM

Yes. Your IDs are safe to post publicly. You should make sure they match. Also make sure your JSON is serialized as you expect.

Matt Silverlock•7/7/24, 10:58 AM

Beyond that, I can’t help more here - we have a lot of critical workflows using the REST API and your JS app is fine, so something is up with your client.

Matt Silverlock•7/7/24, 10:59 AM

My Rust is… rusty, but I suspect you are potentially passing an empty body somehow when o you’re serializing it.

Matt Silverlock•7/7/24, 11:01 AM

(Should .json(body).json(body) instead be .json(&body).json(&body) - ?)

Matt Silverlock•7/7/24, 11:01 AM

Also see https://stackoverflow.com/a/66589310 @Fake User

Stack Overflow

JSON Body in POST Using the Rust reqwest Crate

I'm trying to implement this curl call using the Rust crate reqwest:
curl -u [USERNAME]:[PASSWORD] -H "Content-Type: application/json" -d @content.json [WEBSITE]

The file content.json en...

Matt Silverlock•7/7/24, 11:02 AM

From the documentation for .json() we can see that it expects a arbitrary serde serializable type. This indicates that what it's doing is creating JSON text from the given value. So, you're sending a JSON-quoted string containing JSON to the server, which is presumably not what it's expecting.
I haven't worked with reqwest, but it looks like for your use case with a JSON string, you would use .body() to provide the string and .header() to specify the JSON Content-Type.

… which would describe why you’re likely sending an empty body.

similardisaster•7/8/24, 8:14 PM

what happens if a worker tries to write more than once per second to the same key in KV? For context, we'd like to write to keys only if they don't yet exist. However, due to the eventual consistency, what if high traffic causes multiple writes within the same second at different PoP?

thomasgauvin•7/8/24, 9:08 PM

@similardisaster your writes to KV beyond the one per second limit fail silently, with only the successful write taking effect, in line with last write wins

similardisaster•7/9/24, 7:31 PM

is ctx.waitUntil() suitable for making kv writes without adding latency (blocking) serving a response? Also, is there an estimate we can use for how much added latency reading 1 key from kv per request will add to an existing worker?

similardisaster•7/9/24, 7:33 PM

additionally, when we roll out there will be an initial influx of writes - any concerns with rolling out straight into up to 10k writes/second (mostly to unique keys)?

Hello, I’m Allie!•7/9/24, 7:37 PM

Question #1: Yes
#2: Not really, it depends where in the world you are writing from, since it writes to US/EU.
#3: 10k writes/s should be perfectly manageable assuming, as you pointed out, that it is spread across unique keys as much as possible

HHello, I’m Allie!Question #1: Yes #2: Not really, it depends where in the world you are writing f...

similardisaster•7/9/24, 7:39 PM

thank you!

similardisaster•7/10/24, 2:12 PM

I'm a bit confused on cacheTtl - where is the best place to set it? In each get? Each key is write once with an expirationTtl of 30 days (not even checking the value just checking the keys existence). Should I just do each get with a 30 day cacheTtl? Or would that result in keys returning a cached value even though they actually have since expired?

thomasgauvin•7/10/24, 3:26 PM

Yes, you would set the expirationTtl for each get, they keys will not be returned past expiration

I'm building something like google sheets. Each user could have multiple sheets and I could have tho

Similar Threads

Similar Threads

Similar Threads