Production workers 100% errors

Randomly started getting 500 errors in production env workers

{
  "outcome": "exception",
  "scriptName": "mage-api",
  "diagnosticsChannelEvents": [],
  "exceptions": [
    {
      "name": "Error",
      "message": "The script will never generate a response.",
      "timestamp": 1712089582194
    }
  ],

30 Replies

James•2mo ago

Did you change anything? What does your worker do? Do you have a minimal reproduction?

Gagan Suie•2mo ago

nothing was changed i've tried reverting to previous deployments and redeploying as well what do you mean minimal reproduction? to reproduce the error, just visit our website and any api trigered will give that response

Gagan Suie•2mo ago

https://mage.stream

Mage - A collaborative streaming platform

Mage is a collaborative streaming platform for gamers and developers

James•2mo ago

I mean some code that others can run - a minimal example - to reproduce the issue There's not been any widespread reports I'm aware of, so likely going to require some debug and digging into where your worker is failing

Gagan Suie•2mo ago

i finally figured out the error. it was related to packages. I reverted packages from a few days ago and its working now. Now i need to investigate which package was causing issues.......🙃 these are the working packages:

 "dependencies": {
        "@cloudflare/itty-router-openapi": "^1.0.6",
        "@miniflare/d1": "^2.14.1",
        "@tsndr/cloudflare-worker-jwt": "^2.2.1",
        "itty-router": "^4.0.11",
        "moment": "^2.29.4",
        "stripe": "^14.1.0",
        "tslib": "^2.5.0",
        "worker-auth-providers": "^0.0.13"
    },
    "devDependencies": {
        "@cloudflare/workers-types": "^4.20240208.0",
        "@types/jest": "^29.5.0",
        "@typescript-eslint/eslint-plugin": "7.3.1",
        "@typescript-eslint/parser": "7.2.0",
        "esbuild": "^0.20.0",
        "eslint": "^8.37.0",
        "eslint-config-prettier": "^9.0.0",
        "eslint-config-typescript": "^3.0.0",
        "jest": "^29.5.0",
        "jest-environment-miniflare": "^2.13.0",
        "miniflare": "^3.0.0",
        "prettier": "3.2.5",
        "ts-jest": "^29.1.0",
        "ts-node": "^10.9.1",
        "typescript": "^5.0.3",
        "wrangler": "^3.23.0"
    }

and this version was causing errors:

    "dependencies": {
        "@cloudflare/itty-router-openapi": "^1.0.6",
        "@miniflare/d1": "^2.14.2",
        "@tsndr/cloudflare-worker-jwt": "^2.2.1",
        "itty-router": "^5.0.6",
        "moment": "^2.29.4",
        "stripe": "^14.1.0",
        "tslib": "^2.5.0",
        "worker-auth-providers": "^0.0.13"
    },
    "devDependencies": {
        "@cloudflare/workers-types": "^4.20240329.0",
        "@types/jest": "^29.5.0",
        "@typescript-eslint/parser": "7.5.0",
        "@typescript-eslint/eslint-plugin": "7.5.0",
        "esbuild": "^0.20.0",
        "eslint": "^8.37.0",
        "eslint-config-prettier": "^9.0.0",
        "eslint-config-typescript": "^3.0.0",
        "jest": "^29.5.0",
        "jest-environment-miniflare": "^2.13.0",
        "miniflare": "^3.20240329.0",
        "prettier": "3.2.5",
        "ts-jest": "^29.1.0",
        "ts-node": "^10.9.1",
        "typescript": "^5.0.3",
        "wrangler": "^3.44.0"
    }

James•2mo ago

I would guess itty-router since that's a major bump - did you check the breaking changes? Generally I'd recommend double checking this in future 🙂 Glad you figured it out though!

Gagan Suie•2mo ago

you were right sir. great deduction! -i've never seen this error message and everything on google said it was an infinite loop issue on cloudflares side. And cloudflare had some outages today. Also a few people resolved the issue by reverting some random packages they had. -i did revert to previous deployments, but i guess i didn't go back far enough. -i have mergify and dependabot that automatically created and deployed new package versions (they dont deploy to production until i manually merge the changes). And i guess itty-router slipped under the radar this time!

James•2mo ago

Awesome to hear! You might want to spin up a few tests for your worker using the new vitest integration to catch these kind of things in those automation version bumps in future: https://blog.cloudflare.com/workers-vitest-integration If you're doing automatic dependency bumps, having some basic tests for if your worker still runs is something I would very strongly advise Even if it's just a smoke test that hits / - that would probably have caught this hanging issue from the version bump, and prevented the PR from merging 😄

Gagan Suie•2mo ago

i will investigate this tonight thank you!

Kevin•2mo ago

I definitely recommend the v4 --> v5 Migration Guide: https://itty.dev/itty-router/migrations/v4-v5 But the biggest thing you have to do is to swap any .handle() references to .fetch(), as .handle() is now fully deprecated (v4 allowed both). Otherwise, when you ask for .handle() (no longer found), the Proxy catches it and assumes you're trying to register a route, not handle a request. Mayhem ensues. 🫠

Migration Guides > v4 to v5 - itty-router - itty.dev

Ultra-small, powerful helpers for modern serverless APIs.

Gagan Suie•2mo ago

yep. that was the issue. im looking to actually migrate to @Cloudflare/itty-router-openapi but im having trouble restricting the /docs endpoint

Kevin•2mo ago

That one should be safe (they're still on v4). I'll be helping them get fully onto the new v5 stuff soon 🙂

Gagan Suie•2mo ago

do you know if something like this is allowed? router.get('/docs', async () => await isAdmin(request, env))

Kevin•2mo ago

it depends on what's in isAdmin 🙂 (also, no need for the async/await in there)

Gagan Suie•2mo ago

isAdmin returns true if user is an admin false otherwise

Kevin•2mo ago

assuming isAdmin is something like this:

const isAdmin = ({ headers }) => {
  if (headers.get('Authorization') !== 'foo') {
    throw new StatusError(401, 'Not Authorized')

    // alernatively
    return error(401)
  }

  // no return here allows further handlers
}

const isAdmin = ({ headers }) => {
  if (headers.get('Authorization') !== 'foo') {
    throw new StatusError(401, 'Not Authorized')

    // alernatively
    return error(401)
  }

  // no return here allows further handlers
}

yeah that's the prob... in itty, any handler that returns anything, stops the flow and responds (with whatever that was). The exception is explicitly returning undefined (that's allowed). So instead, you want it to not return anything to pass onto the next handlers/routes, or return (or throw) to break out early. It's an inversion on the express patterns, where you specifically had to allow next handlers to pass. Allows for typically far simpler/shorter handler code, but requires some rethinking (and has some gotchas)

Gagan Suie•2mo ago

the isAdmin middlware code doesn't even hit

    const router = OpenAPIRouter()
    router.get('/docs', () => isAdmin(request, env))

Kevin•2mo ago

If you're trying to cover an entire branch, don't forget the wildcard. that route will only match /docs, not /docs/foo

router.get('/docs/*', isAdmin)

router.get('/docs/*', isAdmin)

Gagan Suie•2mo ago

the swaggerui dashboard is still visible. but now /docs is appearing in swaggerui and when i try to execute it, i get unauthorized.

Kevin•2mo ago

// where is the request, env coming from here?
router.get('/docs', () => isAdmin(request, env)) 

// here isAdmin naturally has access to the request, env, etc passed from router.fetch
router.get('/docs', isAdmin)

// where is the request, env coming from here?
router.get('/docs', () => isAdmin(request, env)) 

// here isAdmin naturally has access to the request, env, etc passed from router.fetch
router.get('/docs', isAdmin)

have a repo for me to take a look at?

Gagan Suie•2mo ago

the frontend is opensource, the backend is not unfortunately

Gagan Suie•2mo ago

but you can watch my live stream on this very platform: https://mage.stream/channel/18

Mage - A collaborative streaming platform

Mage is a collaborative streaming platform for gamers and developers

Gagan Suie•2mo ago

can you see my stream?

Kevin•2mo ago

I can

Gagan Suie•2mo ago

brilliant!

Kevin•2mo ago

was just logging in ok, to put your own upstream catch before OpenAPI:

import { Router, error } from 'itty-router'
import { OpenAPIRouter } from '@cloudflare/itty-router-openapi'

const rootRouter = Router()
const router = OpenAPIRouter()

// example middleware
const isAdmin = (request) => {
  if (!request.headers.get('Authorization')) {
    return error(401)
  }
}

rootRouter
  // we register the auth catch on the root router
  .all('/docs/*', isAdmin)

  // then attach the next router to handle all requests
  .all('*', router.fetch)

export default {
  fetch: router.fetch
}

import { Router, error } from 'itty-router'
import { OpenAPIRouter } from '@cloudflare/itty-router-openapi'

const rootRouter = Router()
const router = OpenAPIRouter()

// example middleware
const isAdmin = (request) => {
  if (!request.headers.get('Authorization')) {
    return error(401)
  }
}

rootRouter
  // we register the auth catch on the root router
  .all('/docs/*', isAdmin)

  // then attach the next router to handle all requests
  .all('*', router.fetch)

export default {
  fetch: router.fetch
}

Gagan Suie•2mo ago

that didn't work either. i learned how to handle it properly via their parameters:

    const router: any = OpenAPIRouter({
        docs_url: env.ENV === 'local' ? '/docs' : 'null',
        redoc_url: env.ENV === 'local' ? '/redocs' : 'null'
    })

i was sending empty string. but it required a null string.

Kevin•2mo ago

oh super weird...

Gagan Suie•2mo ago

hey thanks again for helping me remove a lot of fluff from the routes!

Kevin•2mo ago

Happy to help!

Gaming

Programming

Production workers 100% errors