CD
Cloudflare Developers•3w ago
josula

Caching NextJS app on worker has long KV cache put times

Hey everyone, we're implementing a custom HTML caching layer in a Worker and running into an unexpected blocking issue that seems to contradict the documentation. We're using ctx.waitUntil() for a background KV write, but the operation is blocking the client response. The Problem We're seeing two main issues in our traces: 1. High Latency Write: Our kv_put operation takes about 900ms to write a large HTML document into the cache. 2. Blocking Request: This 900ms latency is blocking the total request time for the client, delaying the return out statement. Our Code & Confusion My understanding is that ctx.waitUntil(Promise) should allow the main fetch handler to return immediately while the Promise resolves in the background. 
    // we forward the request to the actual nextjs worker since its not cached
    let res = await nextWorker.fetch(req, env, ctx)

    if (res.ok && res.body) {
      // First, check for compression and decompress if necessary
      if (res.headers.get("Content-Encoding")) {
        res = new Response(res.body, res)
        res.headers.delete("Content-Encoding")
      }
      // since nextjs streams, we are tee-ing the response to return one part and cache the other copy
      const [toClient, toCache] = res.body.tee()

// We are constructing the response to the client
const out = new Response(toClient, {
    status: res.status,
    statusText: res.statusText,
    headers: clientHeaders,
})

// We perform the background cache insert
ctx.waitUntil(
    this.cacheResponse(env, cacheKey, toCache, res.status, cacheHeaders)
)

// We expect to return to the client immediately here
return out
    // we forward the request to the actual nextjs worker since its not cached
    let res = await nextWorker.fetch(req, env, ctx)

    if (res.ok && res.body) {
      // First, check for compression and decompress if necessary
      if (res.headers.get("Content-Encoding")) {
        res = new Response(res.body, res)
        res.headers.delete("Content-Encoding")
      }
      // since nextjs streams, we are tee-ing the response to return one part and cache the other copy
      const [toClient, toCache] = res.body.tee()

// We are constructing the response to the client
const out = new Response(toClient, {
    status: res.status,
    statusText: res.statusText,
    headers: clientHeaders,
})

// We perform the background cache insert
ctx.waitUntil(
    this.cacheResponse(env, cacheKey, toCache, res.status, cacheHeaders)
)

// We expect to return to the client immediately here
return out
I think the main issue is how tee works and that it awaits both streamed forks of the reply to be finished (or awaited) before it returns. That is why we see these crazy latencies because our caching system then puts it into the KV cache but it seems like it's awaited before we return to the client
No description
1 Reply
josula
josulaOP•2w ago
You can see from the screenshot that the KV put takes around 900 milliseconds to cache the response and blocks the overall return of the request. My assumption is that it somehow has something to do with the tee and also that we await the cache inside the ctx.waitUntil This is what our caching function actually looks like : 
  async cacheResponse(env, key, bodyStream, status, headers) {
    try {
      const headerObject = Object.fromEntries(headers)
      delete headerObject.link

      await env.CACHE_KV.put(key, bodyStream, {
        expirationTtl: STALE_TTL,
        metadata: {
          timestamp: Date.now(),
          ttl: CACHE_TTL * 1000,
          status,
          headers: headerObject,
        },
      })
    } catch (e) {
      // No console.log
    }
  },
  async cacheResponse(env, key, bodyStream, status, headers) {
    try {
      const headerObject = Object.fromEntries(headers)
      delete headerObject.link

      await env.CACHE_KV.put(key, bodyStream, {
        expirationTtl: STALE_TTL,
        metadata: {
          timestamp: Date.now(),
          ttl: CACHE_TTL * 1000,
          status,
          headers: headerObject,
        },
      })
    } catch (e) {
      // No console.log
    }
  },
I am not deep enough into streams and Cloudflare to fully understand the problem and answer the question. But we somehow need to find a solution to stream the server response back to the client while caching the response at the same time without blocking the thread Does anyone know what we are doing wrong or how to optimize this code so that we immediately return to the client and in the background put the HTML response into the cache? Does anyone have any clue?

Did you find this page helpful?