Latency - Expensive calculations in third party api to be cached on server for a certain amount of t

Latency - Expensive calculations in third party api to be cached on server for a certain amount of time

DaniFoldi•3/10/23, 10:01 AM

Bear in mind that it does not replicate between colos, so you might see "more" misses depending on the distribution of visitors - for which you can use KV as a second option (BetterKV does this in https://flareutils.pages.dev)

DDaniFoldi Bear in mind that it does not replicate between colos, so you might see "more" m...

beelzabub•3/10/23, 2:56 PM

BetterKV is fine in terms of reducing your KV billing cost but I’m not a fan of it and it feels like an artifact of our billing than anything else. Is there any other features it has beyond “cache in front of Kv”?

SSly Latency - Expensive calculations in third party api to be cached on server for a...

beelzabub•3/10/23, 2:56 PM

What’s the latency you need?

Bbeelzabub BetterKV is fine in terms of reducing your KV billing cost but I’m not a fan of ...

DaniFoldi•3/10/23, 2:57 PM

Oh absolutely, the main purpose is to remove the cost of cached KV reads, or at least this is what we use it for.

A quote from the website even says

Save on your Workers bill by "manually" caching every response, without having to lift a finger!

DDaniFoldi Oh absolutely, the main purpose is to remove the cost of cached KV reads, or at ...

beelzabub•3/10/23, 3:01 PM

Yeah. I hope to take a look at our billing in Q2 to see what actually makes sense. BetterKV feels technically superfluous and a recipe for subtle bugs at scale.

Bbeelzabub Yeah. I hope to take a look at our billing in Q2 to see what actually makes sens...

DaniFoldi•3/10/23, 3:05 PM

We were actually discussing this with @mrbbot (sorry) last week, whether it would make sense (from a user perspective) to only charge for cold/uncached reads - basically make KV Better

, as now everything would be double-cached to cache at KV-level, and at Worker-level. From what I remember, the argument against was that it would result in less predictable billing, but the counter-argument to that is while less predictable, still bounded by the current number of reads.

DaniFoldi•3/10/23, 3:07 PM

From there (oh you will not like me for this), cache(+pull if stale) reads and force refresh reads could be separated out and billed separately, turning KV into a powerful key-value storage with a different type of push-pull consistency model than what you have today (with KV 2.0).

DDaniFoldi From there (oh you will not like me for this), cache(+pull if stale) reads and f...

beelzabub•3/10/23, 3:07 PM

I don’t know that the problem is unpredictability. And I don’t hate you because that’s very much along the lines of how I’m thinking about it.

Bbeelzabub I don’t know that the problem is unpredictability. And I don’t hate you because...

beelzabub•3/10/23, 3:09 PM

But I do want to keep the API as-is and do the billing transparently. So I’m not sure yet what the new pricing would look like. And I also have 0 concrete idea of what that looks like. Once 2.0 rolls out fully I’ll have data to formulate an actual opinion

Bbeelzabub But I do want to keep the API as-is and do the billing transparently. So I’m not...

beelzabub•3/10/23, 3:10 PM

if it was just up to me, you know I like you guys. I’d just 0-rate everyone on here :D.

Bbeelzabub But I do want to keep the API as-is and do the billing transparently. So I’m not...

DaniFoldi•3/10/23, 3:15 PM

I don't want to call this a problem, but I couldn't come up with any better phrase - so the problem with billing for cached reads is that using Cache API on top (that is, BetterKV) will still be an improvement over the default behaviour (no added latency, but reduced bill).

Unsmart•3/10/23, 3:25 PM

0 rate is nice NODDERS

Unsmart•3/10/23, 3:26 PM

I will say though I do think it is a bit odd to offer a 100% free cache API but then charge for cached reads on KV

Unsmart•3/10/23, 3:27 PM

Durable objects also charge for the in memory cached reads which is odd too NotLikeThis

DaniFoldi•3/10/23, 3:37 PM

Coming from this model originally I believe is fine, as a next step it would be very Cloudflare (ahem, 0-egress MeowHeartCloudflare

, end billing when stream response returned, etc) to now say ok, cached reads are now free

UUnsmart Durable objects also charge for the in memory cached reads which is odd too <:No...

DaniFoldi•3/10/23, 3:40 PM

I didn't even think about DOs, that's another good one - that would eliminate another lru-cachelru-cache (https://www.npmjs.com/package/lru-cache) usage

DDaniFoldi I didn't even think about DOs, that's another good one - that would eliminate an...

Hello, I’m Allie!•3/10/23, 3:52 PM

The issue with building your own cache on top of DO storage is that you don’t have a real easy way of measuring memory usage, where iirc the default in-memory cache auto-evicts when memory is low

DaniFoldi•3/10/23, 3:54 PM

lru-cache can (sort of) deal with that, for ~100 keys that shouldn't be an issue even if all keys are 128KB, but most (at least for me) are under 2KB

beelzabub•3/10/23, 4:10 PM

Yeah. The team is aware of the DO thing. That being said, most providers will do caching underneath of any storage product and eat the savings so it's not rare and I don't think it would change for DO. I wouldn't recommend putting Cache in front of DO because you have no way of guaranteeing the consistency of the contents in cache / DO. lru-cache is a cute idea and would work now but I suspect it'll not be sustainable as DO matures over the years for more than toy examples (although I could be wrong).

Maybe the main actual weirdness is that the Cache API is free. I'll see about making sure we start charging for it. Be right back. Will flip the flag for that.

Unsmart•3/10/23, 4:12 PM

I think the only reason people complain about charged cache reads on CF is the fact that the cache api is free tbh

Bbeelzabub Yeah. The team is aware of the DO thing. That being said, most providers will do...

Hello, I’m Allie!•3/10/23, 4:12 PM

What have you done, Dani?! YOU’VE DOOMED US ALL!!!

Unsmart•3/10/23, 4:12 PM

Doesnt totally make sense in peoples minds that cache api is free but cached database reads are not free

Hello, I’m Allie!•3/10/23, 4:13 PM

Especially if there isn’t a whole lot of extra logic that goes into the cache. For example, you could probably charge a little bit for SWR caching on KV, but maybe not as much as a full read

Hello, I’m Allie!•3/10/23, 4:14 PM

The auto-pull while cached thing is a feature on top of what the Cache API offers, which is why I think you could justify at least a slight cost there

HHello, I’m Allie!*What have you done, Dani?! **YOU’VE DOOMED US ALL!!!***

DaniFoldi•3/10/23, 4:15 PM

We've actually agreed with @beelzabub in DMs that I'd get 1% of all Cache API bills evil_parrot

DDaniFoldi We've actually agreed with @beelzabub in DMs that I'd get 1% of all Cache API bi...

Hello, I’m Allie!•3/10/23, 4:15 PM

I’ll be fine with that arrangement if you pay the Cache bills for everyone, not just yourself

DaniFoldi•3/10/23, 4:16 PM

Look at us complaining about products priced at $0-5/million range compared to saas's which build on these and charge you $40-50/user/month

DDaniFoldi Look at us complaining about products priced at $0-5/million range compared to s...

Hello, I’m Allie!•3/10/23, 4:18 PM

I mean, I don’t think I can complain at all… :winks_in_green_cloud:

HHello, I’m Allie!I’ll be fine with that arrangement if you pay the Cache bills for **everyone**, ...

DaniFoldi•3/10/23, 4:19 PM

also you're a champ, you have a free ent account, ShutUp

HHello, I’m Allie!I mean, I don’t think I can complain at all… :winks_in_green_cloud:

DaniFoldi•3/10/23, 4:19 PM

yea

DaniFoldi•3/10/23, 4:19 PM

(I'll pay yours of course)

DDaniFoldi (I'll pay yours of course)

Hello, I’m Allie!•3/10/23, 4:19 PM

Disables NoCost just for funzies

HHello, I’m Allie!*Disables NoCost just for funzies*

DaniFoldi•3/10/23, 4:20 PM

I know this is getting sortof #off-topic, but I no longer superlike you, mr superchamp

DDaniFoldi We've actually agreed with @beelzabub in DMs that I'd get 1% of all Cache API bi...

beelzabub•3/10/23, 5:31 PM

Monkey paw wish granted. Everyone gets a 1% coupon thanks to @Dani Foldi volunteering to cover 1% of everyone's bill.

Jjondon As per the documentation, the total subrequest limit for unbound is set at 1000....

beelzabub•3/10/23, 5:31 PM

They count. But note it's per request. Not sure why you'd make repeated KV reads within the context of a single request

Bbeelzabub Monkey paw wish granted. Everyone gets a 1% coupon thanks to @Dani Foldi volunte...

DaniFoldi•3/10/23, 5:32 PM

sudo rm -rf

sudo rm -rf

nooo how else do I stop this

DDaniFoldi Look at us complaining about products priced at $0-5/million range compared to s...

beelzabub•3/10/23, 5:33 PM

I thought https://protomaps.com/blog/serverless-maps-now-open-source was interesting.

A web application with 50,000 map loads using the Google Maps API will cost $350 per month.

A similar amount of traffic running through a Protomaps CDN install on Cloudflare, with an average of 20 tiles per session, costs fifty cents in Cloudflare Workers request fees.

Serverless Maps - Now Open Source

Accelerating Protomaps with a CDN for fast and affordable maps

Bbeelzabub They count. But note it's per request. Not sure why you'd make repeated KV reads...

beelzabub•3/10/23, 5:33 PM

@_john_n does that answer your question? I'm kind of curious about the context where you're concerned about the 1k limit

Bbeelzabub I thought https://protomaps.com/blog/serverless-maps-now-open-source was interes...

Hello, I’m Allie!•3/10/23, 5:33 PM

If you want to see an (admittedly bad) implementation of that, you can try https://maps.goalastair.com

Hello, I’m Allie!•3/10/23, 5:34 PM

Is there an easy way to cache Range Requests? Then I could maybe speed it up a little...

HHello, I’m Allie!Is there an easy way to cache Range Requests? Then I could maybe speed it up a l...

beelzabub•3/10/23, 5:35 PM

I thought range requests are generally cacheable, no? You might need to store the actual asset in cache once but after that cache.match will honor ranges? I forget the details though. CDN behavior is extremely complex and I don't pretend to know it outside of the specific slice of how I'm using it

Bbeelzabub I thought range requests are generally cacheable, no? You might need to store th...

Hello, I’m Allie!•3/10/23, 5:38 PM

I don't think so? I tried tailing the Function, and I'm getting

GET https://maps.goalastair.com/planet - Ok @ 3/10/2023, 6:38:19 PM
✘ [ERROR]   TypeError: Cannot cache response to a range request (206 Partial Content).

GET https://maps.goalastair.com/planet - Ok @ 3/10/2023, 6:38:19 PM
✘ [ERROR]   TypeError: Cannot cache response to a range request (206 Partial Content).

kian•3/10/23, 5:44 PM

put

put

will want the full file but you can make ranged reads against

match

match

kian•3/10/23, 5:45 PM

Same-ish behaviour as the normal CDN, except that all happens transparently (it'll pull the full file and serve you the range, then serving other ranged reads from that full file)

Kkian `put` will want the full file but you can make ranged reads against `match`

Hello, I’m Allie!•3/10/23, 5:46 PM

Ok, so there goes my idea. No way am I going to be able to

put

put

a 60 GB file in every requested colo…

kian•3/10/23, 5:47 PM

You could probably hack around it, emphasis on hack.

kian•3/10/23, 5:48 PM

put

put

just doesn't like the status code of 206 - but if you store a ranged response w/ a body of

then you'll be matching any ranged read against that unless you store the requested range as apart of the cache key.

Latency - Expensive calculations in third party api to be cached on server for a certain amount of t

Similar Threads

Similar Threads

Similar Threads