@Aditya Mathur are you sure you meant to use `expiration` (seconds since the epoch) or `expirationTt

@Aditya Mathur are you sure you meant to use

expiration

expiration

(seconds since the epoch) or

expirationTtl

expirationTtl

(seconds from now) ? Because I saw your

this.windowMs = 60_000 // 1 minute

this.windowMs = 60_000 // 1 minute

comment, but you also have

expirationTtl: this.windowMs

expirationTtl: this.windowMs

and that would be wrong; it would be 60000 seconds.

IIan @Aditya Mathur are you sure you meant to use `expiration` (seconds since the epo...

Aditya Mathur•7/2/24, 5:13 AM

Yeah, you are right about that but it is commented, so it is not getting executed. This is a rate limiter, so the problem is every time a user requests the server I have to update the

totalHits

totalHits

but the expiration time remains that same, thats why it is stored in

resetTime

resetTime

. So when I am updating it, I am passing the

expiration

expiration

again.

IanOP•7/2/24, 5:14 AM

In the docs

As of January 2022, expiration targets that are less than 60 seconds into the future are not supported. This is true for both expiration methods.

IanOP•7/2/24, 5:16 AM

@Aditya Mathur when you read the KV, doesn't it also contain the

expiration

expiration

attribute?

IanOP•7/2/24, 5:17 AM

If so, just merge a default payload with a payload that it found...

Aditya Mathur•7/2/24, 5:18 AM

So what you are saying is, by updating the entry I am not overwriting the

expiration

expiration

IanOP•7/2/24, 5:19 AM

yeah, don't bother updating expiration since it's basically "expires at Time", while the expirationTtl is "expires in X"

IanOP•7/2/24, 5:20 AM

expiration is relative to a fixed point in time, the epoch. while expirationTtl is relative to right now.

IanOP•7/2/24, 5:21 AM

so do this

have a default payload
get the KV
merge it with the default payload using spread operations
put the KV

Aditya Mathur•7/2/24, 5:22 AM

Okay let me try this

AAditya Mathur Okay let me try this

IanOP•7/2/24, 5:24 AM

Something like this

const defaultPayload = {
  hits: 1,
  expiration: new Date(new Date().getTime() + 60_000),
}

const record = await env.NAMESPACE.get('key', { type: 'json' })

const payload = {
  ...defaultPayload,
  ...(record ?? {}), // if record is nullish, spread an empty object in its place
}

await env.NAMESPACE.put('key', payload, { expiration: payload.expiration })

const defaultPayload = {
  hits: 1,
  expiration: new Date(new Date().getTime() + 60_000),
}

const record = await env.NAMESPACE.get('key', { type: 'json' })

const payload = {
  ...defaultPayload,
  ...(record ?? {}), // if record is nullish, spread an empty object in its place
}

await env.NAMESPACE.put('key', payload, { expiration: payload.expiration })

IIan Something like this ```typescript const defaultPayload = { hits: 1, expirati...

Aditya Mathur•7/2/24, 5:35 AM

No Luck

Aditya Mathur•7/2/24, 5:36 AM

Here is the code -

async increment(key: string): Promise<ClientRateLimitInfo> {
    const keyWithPrefix = this.prefixKey(key);

    const defaultPayload = {
      totalHits: 1,
      resetTime: new Date(Date.now() + this.windowMs),
    }

    // @ts-ignore
    const record: Required<ClientRateLimitInfo> | null = await this.namespace.get<Required<ClientRateLimitInfo>>(
      keyWithPrefix,
      "json",
    );

    const payload = { ...defaultPayload, ...(record ? { totalHits: record.totalHits + 1, resetTime: new Date(record.resetTime) } : {}) }

    console.log("Payload - ", payload, " Expire - ", payload.resetTime.toLocaleString(), " Current - ", new Date().toLocaleString())

    await this.namespace.put(keyWithPrefix, JSON.stringify(payload), {
      expirationTtl: !record ? payload.resetTime.getTime() / 1000 : undefined,
    });

    return payload;
  }

async increment(key: string): Promise<ClientRateLimitInfo> {
    const keyWithPrefix = this.prefixKey(key);

    const defaultPayload = {
      totalHits: 1,
      resetTime: new Date(Date.now() + this.windowMs),
    }

    // @ts-ignore
    const record: Required<ClientRateLimitInfo> | null = await this.namespace.get<Required<ClientRateLimitInfo>>(
      keyWithPrefix,
      "json",
    );

    const payload = { ...defaultPayload, ...(record ? { totalHits: record.totalHits + 1, resetTime: new Date(record.resetTime) } : {}) }

    console.log("Payload - ", payload, " Expire - ", payload.resetTime.toLocaleString(), " Current - ", new Date().toLocaleString())

    await this.namespace.put(keyWithPrefix, JSON.stringify(payload), {
      expirationTtl: !record ? payload.resetTime.getTime() / 1000 : undefined,
    });

    return payload;
  }

Aditya Mathur•7/2/24, 5:59 AM

Is there a working example of using

WorkerKV

WorkerKV

with

hono

hono

which uses

expiration

expiration

expirationTtl

expirationTtl

AAditya Mathur Is there a working example of using `WorkerKV` with `hono` which uses `expiratio...

IanOP•7/2/24, 6:14 AM

If you're using hono just use https://github.com/rhinobase/hono-rate-limiter

GitHub

GitHub - rhinobase/hono-rate-limiter: Rate Limit middleware for Hon...

Rate Limit middleware for Hono Web Server . Contribute to rhinobase/hono-rate-limiter development by creating an account on GitHub.

IIan If you're using hono just use https://github.com/rhinobase/hono-rate-limiter

Aditya Mathur•7/2/24, 6:14 AM

That's interesting, I am the creator of that lib

AAditya Mathur That's interesting, I am the creator of that lib 🤣

IanOP•7/2/24, 6:15 AM

haha, so you are

Aditya Mathur•7/2/24, 6:15 AM

I have added support for cloudflare and the

WorkerKV

WorkerKV

is giving me some trouble

AAditya Mathur I have added support for cloudflare and the `WorkerKV` is giving me some trouble

IanOP•7/2/24, 6:17 AM

I'm curious why you're storing the

resetTime

resetTime

if there's already an

expiration

expiration

field?

IIan I'm curious why you're storing the `resetTime` if there's already an `expiration...

Aditya Mathur•7/2/24, 6:20 AM

This is what I have concluded, we need to update the

expiration

expiration

when we update the value, if you not it's not gonna expire.

AAditya Mathur This is what I have concluded, we need to update the `expiration` when we update...

IanOP•7/2/24, 6:21 AM

Sure, but when you

get('key')

get('key')

it returns an object with the

expiration

expiration

field, right? So if it already has that field, you can re-write it back without altering it.

IanOP•7/2/24, 6:22 AM

When you're altering it, you're taking the

expiration

expiration

which is fixed in time, and keep adding to it each time the request happens. is that what you want? for the expiration to be moved forward?

Aditya Mathur•7/2/24, 6:22 AM

If you update the expiration on each request, you will then be putting values less than 60 seconds which will not be getting registered

Aditya Mathur•7/2/24, 6:22 AM

All I am concluding from this is WorkerKV cannot be used for rate limiting in general because the API is not build for this purpose. It is more for having global config for env or feature flags

IanOP•7/2/24, 6:22 AM

it could be; I have a lateral solution...

Aditya Mathur•7/2/24, 6:23 AM

I am listening

IanOP•7/2/24, 6:23 AM

make a record for each request with a ttl of 15 minutes.

IanOP•7/2/24, 6:23 AM

then list and count all the records on each request to see if they exceed the limit.

IanOP•7/2/24, 6:24 AM

the maximum number of records you'll have is the

limit

limit

. so not very many.

IanOP•7/2/24, 6:24 AM

and you never have to write to them after the first

put()

put()

; you fire and foget them until the next request where you

list()

list()

and count.

IanOP•7/2/24, 6:25 AM

basically use the namespace as a "leaky bucket"

IIan then list and count all the records on each request to see if they exceed the li...

Aditya Mathur•7/2/24, 6:26 AM

Yes you can manually work this out i.e. without the use of

expiration

expiration

prop. But at the end of the day it will be too expensive to use this solution for rate limiting.

IIan and you never have to write to them after the first `put()`; you fire and foget ...

Aditya Mathur•7/2/24, 6:27 AM

Because the limit is 1,000/day. Check this https://developers.cloudflare.com/kv/platform/pricing/

Cloudflare Docs

Pricing · Cloudflare Workers KV

Workers KV is included in both the Free and Paid Workers plans.

IanOP•7/2/24, 6:29 AM

You could use D1 https://developers.cloudflare.com/d1/platform/pricing/

Cloudflare Docs

Pricing · Cloudflare D1 docs

D1 bills based on:

IanOP•7/2/24, 6:29 AM

There's also durable objects.

IIan There's also durable objects.

Aditya Mathur•7/2/24, 6:29 AM

Yes, that is what I am thinking about to drop the support for

WorkerKV

WorkerKV

and use

DurableObjects

DurableObjects

AAditya Mathur Yes, that is what I am thinking about to drop the support for `WorkerKV` and use...

IanOP•7/2/24, 6:31 AM

the kv plan for paid customers still makes sense.

AAditya Mathur Yes, that is what I am thinking about to drop the support for `WorkerKV` and use...

IanOP•7/2/24, 6:33 AM

https://developers.cloudflare.com/durable-objects/examples/build-a-rate-limiter/ here's an actual example of building a rate limiter with durable objects

IIan https://developers.cloudflare.com/durable-objects/examples/build-a-rate-limiter/...

Aditya Mathur•7/2/24, 6:34 AM

Nice, will be building something like this then

IanOP•7/2/24, 6:35 AM

https://community.cloudflare.com/t/creating-a-rate-limiter-using-workers-kv/319256 there is also this forum post about using WorkerKV for rate limiting

IanOP•7/2/24, 6:36 AM

KV is not really a good idea here due to it’s eventually consistent nature. You can however make a rate-limit from Durable Objects 272, which some others have already made from too.

IIan > KV is not really a good idea here due to it’s eventually consistent nature. Yo...

Aditya Mathur•7/2/24, 6:36 AM

Yeah

Aditya Mathur•7/2/24, 6:37 AM

Now I know why is that

vyastj•7/2/24, 10:05 AM

Hello

Can someone help me here or point to correct channel ?

workers-ai

Xiangmian•7/3/24, 7:42 AM

I can't log in either

LLawrence Hi, wrangler login always got error, did anybody knows how to fix that? ``` Synt...

Bọc Link VN•7/3/24, 8:52 AM

I got the same today when use npx wrangler login

LLawrence Hi, wrangler login always got error, did anybody knows how to fix that? ``` Synt...

ivxenog•7/3/24, 9:17 AM

same here.

Hard@Work•7/3/24, 9:43 AM

@twodft @Xiangmian @crypto Howdy! The team is aware, and they are working on a fix

Vero•7/3/24, 2:15 PM

Hey. This should be fixed now, can you please confirm?

Ross•7/3/24, 4:29 PM

npx wrangler pages deploy

npx wrangler pages deploy

is erroring for me, Cloudflare having API issues?

 ✘ [ERROR] Received a malformed response from the API

  upstream connect error or disconnect/reset before headers. retried and the latest reset reason: conn... (length = 114)
  GET /accounts/***/pages/projects/example -> 503 Service Unavailable

 ✘ [ERROR] Received a malformed response from the API

  upstream connect error or disconnect/reset before headers. retried and the latest reset reason: conn... (length = 114)
  GET /accounts/***/pages/projects/example -> 503 Service Unavailable

const defaultPayload = { hits: 1, expiration: new Date(new Date().getTime() + 60_000), } const record = await env.NAMESPACE.get('key', { type: 'json' }) const payload = { ...defaultPayload, ...(record ?? {}), // if record is nullish, spread an empty object in its place } await env.NAMESPACE.put('key', payload, { expiration: payload.expiration })

async increment(key: string): Promise<ClientRateLimitInfo> { const keyWithPrefix = this.prefixKey(key); const defaultPayload = { totalHits: 1, resetTime: new Date(Date.now() + this.windowMs), } // @ts-ignore const record: Required<ClientRateLimitInfo> | null = await this.namespace.get<Required<ClientRateLimitInfo>>( keyWithPrefix, "json", ); const payload = { ...defaultPayload, ...(record ? { totalHits: record.totalHits + 1, resetTime: new Date(record.resetTime) } : {}) } console.log("Payload - ", payload, " Expire - ", payload.resetTime.toLocaleString(), " Current - ", new Date().toLocaleString()) await this.namespace.put(keyWithPrefix, JSON.stringify(payload), { expirationTtl: !record ? payload.resetTime.getTime() / 1000 : undefined, }); return payload; }

✘ [ERROR] Received a malformed response from the API upstream connect error or disconnect/reset before headers. retried and the latest reset reason: conn... (length = 114) GET /accounts/***/pages/projects/example -> 503 Service Unavailable

@Aditya Mathur are you sure you meant to use `expiration` (seconds since the epoch) or `expirationTt

Similar Threads