CD
Cloudflare Developers•3w ago
S̷̄k̷̎̈́ẏ̵̚ŵ̶͠ǒ̶l̵̓͝f̶̂4̴͘6

Worker force exit when HyperDrive high concurrency connection

Hello, I'm try to use cloudflare features I didn't experimented, and reached to HyperDrive. Currently, my service is growing very fast, so getting out of free tier, keep draining my budget double at month - so I decided to try HyperDrive instead of D1. It works great at first time, but with 1 hours of monitering I see very big problem - HyperDrive connection occurs worker timeout with code force exit. I don't know why it is, but guessing it's related to high amount of concurrent database connection. My service is getting average 10 request per second, and this means I need 30+ concurrent connection at traffic burst timing. How can I deal it with HyperDrive, can anyone give some advice to me? Using PosgreSQL + TimescaledDB + HyperDrive for real-time analysis. Getting this error for now. 
The Workers runtime canceled this request because it detected that your Worker's code had hung and would never generate a response. Refer to: https://developers.cloudflare.com/workers/observability/errors/"
The Workers runtime canceled this request because it detected that your Worker's code had hung and would never generate a response. Refer to: https://developers.cloudflare.com/workers/observability/errors/"
Using this test code: 
  private static async appendLog(c: DrizzleContext, image: ImageType) {
    try {
      await GeneralHyperdrive.getOrCreateDrive(
        "log",
        c.context.env.SUB_DATABASE
      )
        .db()
        .insert(LOG_TABLE)
        .values({
          time: new Date(),
          articleid: image.articleId,
          traffic: image.fileSize,
          referer:
            c.context.req.header("Referer") ?? c.context.req.header("referer"),
        });
    } catch (e) {}
  }

  // class GenericHyperdrive
  static getOrCreateDrive = function (
    name: string,
    drive: Hyperdrive | undefined = undefined
  ) {
    if (GeneralHyperdrive.drives.has(name)) {
      return GeneralHyperdrive.drives.get(name);
    }

    const pool = new Pool({
      connectionString: drive.connectionString,
    });
    const db = drizzle({ client: pool });
    const newDrive = new GeneralHyperdrive(db);

    return newDrive;
  };
  private static async appendLog(c: DrizzleContext, image: ImageType) {
    try {
      await GeneralHyperdrive.getOrCreateDrive(
        "log",
        c.context.env.SUB_DATABASE
      )
        .db()
        .insert(LOG_TABLE)
        .values({
          time: new Date(),
          articleid: image.articleId,
          traffic: image.fileSize,
          referer:
            c.context.req.header("Referer") ?? c.context.req.header("referer"),
        });
    } catch (e) {}
  }

  // class GenericHyperdrive
  static getOrCreateDrive = function (
    name: string,
    drive: Hyperdrive | undefined = undefined
  ) {
    if (GeneralHyperdrive.drives.has(name)) {
      return GeneralHyperdrive.drives.get(name);
    }

    const pool = new Pool({
      connectionString: drive.connectionString,
    });
    const db = drizzle({ client: pool });
    const newDrive = new GeneralHyperdrive(db);

    return newDrive;
  };
No description
8 Replies
S̷̄k̷̎̈́ẏ̵̚ŵ̶͠ǒ̶l̵̓͝f̶̂4̴͘6
S̷̄k̷̎̈́ẏ̵̚ŵ̶͠ǒ̶l̵̓͝f̶̂4̴͘6OP•3w ago
I'm very sure this problem came from HyperDrive, because after test deploy error increased.
S̷̄k̷̎̈́ẏ̵̚ŵ̶͠ǒ̶l̵̓͝f̶̂4̴͘6
S̷̄k̷̎̈́ẏ̵̚ŵ̶͠ǒ̶l̵̓͝f̶̂4̴͘6OP•3w ago
Okay, I think I finally got this error : max connection was the problem. After I manually destroy query pool, all errors gone and inner peace came to me. The problem was, node-postgres didn't close the fully connection pool after worker end signal called. I found this issue from here, but looks like it's not fixed and undocumented - Cloudflare documents need to mention close pool connection after task completed. HyperDrive is great thing, but it needs more detailed document. You can see it from example code, document don't closing connection after task completed - it'll be cause big problem when more than 5 req/s.
No description
S̷̄k̷̎̈́ẏ̵̚ŵ̶͠ǒ̶l̵̓͝f̶̂4̴͘6
S̷̄k̷̎̈́ẏ̵̚ŵ̶͠ǒ̶l̵̓͝f̶̂4̴͘6OP•3w ago
Okay fine, It was omitted document from update.
I found HyperDrive document from dashboard is closed correctly:
No description
S̷̄k̷̎̈́ẏ̵̚ŵ̶͠ǒ̶l̵̓͝f̶̂4̴͘6
S̷̄k̷̎̈́ẏ̵̚ŵ̶͠ǒ̶l̵̓͝f̶̂4̴͘6OP•3w ago
But the document at cloudflare docs are outdated. Maybe I'll to pull request later, but not for now due to short time.
No description
flashblaze
flashblaze•2w ago
where is this from? i don't see it in my dashboard
S̷̄k̷̎̈́ẏ̵̚ŵ̶͠ǒ̶l̵̓͝f̶̂4̴͘6
S̷̄k̷̎̈́ẏ̵̚ŵ̶͠ǒ̶l̵̓͝f̶̂4̴͘6OP•2w ago
It's from "Workers & Pages" modal. You can see it from "Add bindings" button.
S̷̄k̷̎̈́ẏ̵̚ŵ̶͠ǒ̶l̵̓͝f̶̂4̴͘6
S̷̄k̷̎̈́ẏ̵̚ŵ̶͠ǒ̶l̵̓͝f̶̂4̴͘6OP•2w ago
No description
S̷̄k̷̎̈́ẏ̵̚ŵ̶͠ǒ̶l̵̓͝f̶̂4̴͘6
S̷̄k̷̎̈́ẏ̵̚ŵ̶͠ǒ̶l̵̓͝f̶̂4̴͘6OP•2w ago
No description

Did you find this page helpful?