Cloudflare Developers•5mo ago

You know what, that's perfectly reasonable. Let me see if I can knock this one out for our next rele

You know what, that's perfectly reasonable. Let me see if I can knock this one out for our next release.

AAJR You know what, that's perfectly reasonable. Let me see if I can knock this one o...

Focux•8/13/25, 9:55 PM

Thanks so much! I’d really appreciate it!

zsilbi•8/14/25, 10:34 AM

Hello!

I've started to experiment with Hyperdrive +

mysql2

mysql2

but I've ran into memory leak problems just by running the

SHOW TABLES

SHOW TABLES

example from the Hyperdrive tutorial (https://developers.cloudflare.com/hyperdrive/examples/connect-to-mysql/#mysql2)

I've created a new Nitro

eventHandler

eventHandler

with the example code and ran the it 25 times and noticed that the

Connection

Connection

objects are not collected by GC.

Interestingly, if change the the query to only run a single

SELECT

SELECT

on a table, the connection is closed and collected by GC.

I tried to run more complex queries using

drizzle

drizzle

and had the same issue.

Zzsilbi Hello! I've started to experiment with Hyperdrive + `mysql2` but I've ran into ...

AJROP•8/14/25, 11:22 AM

Hi!

I'm not familiar with Nitro at all, so I'm going to have to ask what are likely dumb questions. Please bear with me.

Is this running on a Cloudflare Worker, or some other platform?
Do you see this behavior when using
```
mysql2
```
```
mysql2
```
without Hyperdrive? Or is this only visible when using both together?

zsilbi•8/14/25, 11:26 AM

I build the app using Nitro and run it using

wrangler dev .output/server/index.mjs --assets .output/public --env development --remote

wrangler dev .output/server/index.mjs --assets .output/public --env development --remote

I haven't tested it without Hyperdrive yet, that's what I am going to do now.

AJROP•8/14/25, 11:28 AM

Ah. Local dev does not behave like production Hyperdrive today. It's basically just a TCP connect to your local database. It's a known challenge and we (Workers teams in general) have some roadmap items to improve it.

I expect, then, that you'll see the same behavior without Hyperdrive.

zsilbi•8/14/25, 11:29 AM

As I mentioned, the interesting thing is that if change

SHOW TABLES;

SHOW TABLES;

to just a

SELECT * FROM my_table;

SELECT * FROM my_table;

it works as expected. Each request has a new Connection which is cleared up in

waitUntil

waitUntil

AJROP•8/14/25, 11:31 AM

That is interesting. We're collecting feedback on MySQL in our ecosystem, and the mysql2 maintainers have been very willing to work with us to improve support on our platform, so even if it's driver-only behavior it's still very good to know.

AJROP•8/14/25, 11:31 AM

Also, for what it's worth, in production the Worker runtime will forcibly garbage-collect those, even if the driver itself leaves them open once the request finishes.

zsilbi•8/14/25, 11:35 AM

Thanks for the info, I'll do more tests to collect more data about this.
I'll also try this in staging to see if I can trigger an out of memory situtation.

AJROP•8/14/25, 11:36 AM

Thank you! Looking forward to hearing what you see.

zsilbi•8/14/25, 11:47 AM

I ran it without Hyperdrive and the result is the same as expected.
I also built the app for

node

node

and tested the same event handler. The GC cleared the

Connection

Connection

after each request, so no issues there.
So this problem is not HD related at all.

AJROP•8/14/25, 1:43 PM

Thanks for double-checking, much appreciated!

zsilbi•8/14/25, 2:28 PM

I did some testing with a deployed worker.
I used

FinalizationRegistry

FinalizationRegistry

and

WeakRef

WeakRef

to track the number of existing connections with

console.log

console.log

wrangler tail

wrangler tail

and it seems like the GC never clears them after the request. I tried this also with the

SELECT * FROM my_table

SELECT * FROM my_table

query too which worked in dev and nothing.

After the connection count reaches around 250-300 I just get a new worker and the number drops to 0.

Zzsilbi I did some testing with a deployed worker. I used `FinalizationRegistry` and `We...

AJROP•8/14/25, 2:30 PM

Sorry, not sure I'm following. A "new worker" is assigned for each request.

AAJR Sorry, not sure I'm following. A "new worker" is assigned for each request.

zsilbi•8/14/25, 2:32 PM

import { createConnection, type Connection } from "mysql2/promise";

const registry = new FinalizationRegistry((value) => {
  console.log(`"${value}" was GC'd`);
});

console.log("FinalizationRegistry created");

const connections: WeakRef<Connection>[] = [];

export default defineEventHandler(async (event) => {
  const { env } = event.context.cloudflare;

  const connection = await createConnection({
    host: env.HYPERDRIVE.host,
    user: env.HYPERDRIVE.user,
    password: env.HYPERDRIVE.password,
    database: env.HYPERDRIVE.database,
    port: env.HYPERDRIVE.port,
    disableEval: true
  });

  try {
    registry.register(connection, "MySQL2 connection");
    connections.push(new WeakRef(connection));

    await connection.query("SELECT * FROM `my_table`;");

    event.waitUntil(connection.end());

    return Response.json({
      connections: connections.filter((ref) => ref.deref() !== undefined).length
    });
  } catch (e) {
    console.error(e);
  }
});

import { createConnection, type Connection } from "mysql2/promise";

const registry = new FinalizationRegistry((value) => {
  console.log(`"${value}" was GC'd`);
});

console.log("FinalizationRegistry created");

const connections: WeakRef<Connection>[] = [];

export default defineEventHandler(async (event) => {
  const { env } = event.context.cloudflare;

  const connection = await createConnection({
    host: env.HYPERDRIVE.host,
    user: env.HYPERDRIVE.user,
    password: env.HYPERDRIVE.password,
    database: env.HYPERDRIVE.database,
    port: env.HYPERDRIVE.port,
    disableEval: true
  });

  try {
    registry.register(connection, "MySQL2 connection");
    connections.push(new WeakRef(connection));

    await connection.query("SELECT * FROM `my_table`;");

    event.waitUntil(connection.end());

    return Response.json({
      connections: connections.filter((ref) => ref.deref() !== undefined).length
    });
  } catch (e) {
    console.error(e);
  }
});

I see

FinalizationRegistry created

FinalizationRegistry created

message in the log once during my initial request and after the connection count reaches around 250 to 300.

AJROP•8/14/25, 2:34 PM

Oh I see. That's so interesting, I'm not familiar with that.

Do you have connection monitoring on your origin database? How many actually get opened/held there?

Hyperdrive is kind of unique in that the client connection from the Worker is completely separate from the origin connection to the DB. The client connections are very cheap, and the Hyperdrive+Worker Runtime manage their lifecycle automatically.

Isaac McFadyen•8/14/25, 2:36 PM

I see FinalizationRegistry created message in the log once during my initial request

Also just an FYI that this is expected. The Workers runtime creates one isolate per metal/server (*currently) and TCP connections are sticky so you'll be routed to the same metal for a good bit after you make the initial request. The global scope is only run once at isolate startup so you'll only see that message once.

Isaac McFadyen•8/14/25, 2:37 PM

Also generally why it's not recommended to put anything in the global scope if you can help it

Isaac McFadyen•8/14/25, 2:39 PM

Looking at that code again, I wonder if perhaps the connection is getting closed immediately, as expected because of the

.end()

.end()

, but GC happens later.

Isaac McFadyen•8/14/25, 2:39 PM

Which would imply a connection being closed doesn't GC it immediately.

zsilbi•8/14/25, 2:40 PM

My problem is that the Mysql2 Connection object stays in memory. If I create a new drizzle instance with a huge schema for each Connection these are going to be kept in memory too.

Zzsilbi My problem is that the Mysql2 Connection object stays in memory. If I create a n...

Isaac McFadyen•8/14/25, 2:41 PM

If I create a new drizzle instance with a huge schema for each Connection they are going to be kept in memory too.

Sure, but the GC collector will be run more often if you're near the memory limit, so theoretically you'd see those connections being cleaned up more frequently if they were an issue. But I expect they're super lightweight anyway if the connection itself is closed.

Isaac McFadyen•8/14/25, 2:41 PM

~~Your heap snapshot above showed 0.1KB, that's very very little.~~ Ah wait I see the other number. But I think this still applies ^

zsilbi•8/14/25, 2:42 PM

Yep, that's for the

SHOW TABLES

SHOW TABLES

query only

AAJR Oh I see. That's so interesting, I'm not familiar with that. Do you have connec...

zsilbi•8/14/25, 2:54 PM

The test DB is on PlanetScale and the VTGate says the max. was 21 active connections while I tested.

IIsaac McFadyen > If I create a new drizzle instance with a huge schema for each Connection they...

zsilbi•8/14/25, 3:11 PM

Using only

SHOW TABLES;

SHOW TABLES;

query the max number of connection objects I see is around 360. If I multiply that by the 256 kB retained size I get around 92 MB. If I add the default heap size of the app which is around 36 MB I get 128 MB. Is this a coincidence?

I don't see OOM errors in the CF dashboard for this worker.

Zzsilbi Using only `SHOW TABLES;` query the max number of connection objects I see is ar...

Isaac McFadyen•8/14/25, 3:12 PM

Very well might be intended, I'm not sure how often the GC runs on workerd. But as long as you're not seeing errors it should be fine, the collector should clean up connections when closed.

Isaac McFadyen•8/14/25, 3:12 PM

It's possible the GC is tuned to stay inactive until you're near the heap limit, not sure.

Zzsilbi Using only `SHOW TABLES;` query the max number of connection objects I see is ar...

zsilbi•8/14/25, 3:34 PM

I assigned each connection a UInt8Array with 16 * 1024 * 1024 of "0" and repeated the test. Now the max I get is 12 connections. So this confirms that this is memory related.

zsilbi•8/14/25, 4:23 PM

Ok I found a hack to allow the Connection objects to be GCd:

event.waitUntil(
  connection.end().then(() => {
    (connection as any).connection.stream.destroy();
  })
);

event.waitUntil(
  connection.end().then(() => {
    (connection as any).connection.stream.destroy();
  })
);

I still see no console.log from the

FinalizationRegistry

FinalizationRegistry

but I think that's a platform limitation.
But this way I see only one "FinalizationRegistry created" message and the connections drop to 0 periodically.

Zzsilbi Ok I found a hack to allow the Connection objects to be GCd: ```ts event.waitUn...

Isaac McFadyen•8/14/25, 5:39 PM

I honestly think you do not have to worry about this at all. If the heap limit is neared then

workerd

workerd

will run a GC sweep and clean things up.

Isaac McFadyen•8/14/25, 5:40 PM

The reason you periodically see the

FinalizationRegistry created

FinalizationRegistry created

is probably because you just get routed to a different server and a new isolate is spun up.

Zzsilbi I assigned each connection a UInt8Array with 16 * 1024 * 1024 of "0" and repeate...

zsilbi•8/14/25, 5:47 PM

This confirmed that it's purely memory related. The

Socket

Socket

within mysql2

PromiseConnection.connection.stream

PromiseConnection.connection.stream

prevents it from being cleared.

zsilbi•8/14/25, 5:49 PM

After the fix I ran a test with 5000 connections and it was executed within the same isolate. (saw no console.log from the global scope)

ella•8/15/25, 2:38 PM

does anyone know why i cant add my database even though it has sslmode set to require?

  Failed to connect to the provided database: TLS handshake failed
  [SSLV3_ALERT_HANDSHAKE_FAILURE] [HANDSHAKE_FAILURE_ON_CLIENT_HELLO] [code: 2015]

  Failed to connect to the provided database: TLS handshake failed
  [SSLV3_ALERT_HANDSHAKE_FAILURE] [HANDSHAKE_FAILURE_ON_CLIENT_HELLO] [code: 2015]

Eella does anyone know why i cant add my database even though it has sslmode set to re...

AJROP•8/15/25, 6:03 PM

Local dev, or deployed?

AAJR Local dev, or deployed?

ella•8/15/25, 6:14 PM

well im getting the same error in the dashboard, when i run

npx wrangler hyperdrive create testingdb --connection-string="postgres://postgres:pgpassword@iphere:5432/postgres"

npx wrangler hyperdrive create testingdb --connection-string="postgres://postgres:pgpassword@iphere:5432/postgres"

Eella does anyone know why i cant add my database even though it has sslmode set to re...

AJROP•8/15/25, 6:50 PM

Looks like you're probably running some custom curve for TLS? On Coolify, maybe?

AAJR Looks like you're probably running some custom curve for TLS? On Coolify, maybe?

ella•8/15/25, 7:10 PM

i am using coolify

Eella i am using coolify

AJROP•8/15/25, 7:35 PM

Assuming you're using their automatically configured SSL, it is not currently supported on Hyperdrive. See the below issue for details.

https://github.com/cloudflare/workers-sdk/issues/8671

GitHub

🐛 BUG: Cannot connect to Hyperdrive from Coolify PostgreSQL Data...

Which Cloudflare product(s) does this pertain to? Other What versions are you using? 4.4.0 [Wrangler], 16.8 [PostgreSQL] What operating system and version are you using? Windows 10 21H2 (Wrangler) ...

AAJR Assuming you're using their automatically configured SSL, it is not currently su...

ella•8/15/25, 7:39 PM

what curves are supported?

ella•8/15/25, 7:39 PM

would prime256 be suitable?

AJROP•8/15/25, 7:40 PM

Currrently we support the default curves on pg 14-17. So, ssl_ecdh_curve=prime256v1ssl_ecdh_curve=prime256v1 is supported. Assuming that's the one you mean, yes it works.

letincho5•8/18/25, 1:16 PM

My project uses Drizzle + Hyperdrive + Neon. The last week start to appears this message: "Timed out while waiting for a message from the origin database". I think is a Hyperdrive message. Are there any place to see hyperdrive logs?

Hyperdrive is configured with the pool's connection string of Neon, is it right?
I've trierd Drizzle client with and without a PgNeon Pool --> in both cases, some time I get the error (is not deterministic)
Any ideas?

Lletincho5 My project uses Drizzle + Hyperdrive + Neon. The last week start to appears this...

AJROP•8/18/25, 1:51 PM

Hello. Confirmed, that's a Hyperdrive error message. It is thrown when a query is sent to the origin database, but no response is received for 60 seconds.

Lletincho5 My project uses Drizzle + Hyperdrive + Neon. The last week start to appears this...

AJROP•8/18/25, 1:53 PM

This timeout was added about a month ago. If you are seeing it regularly, I can adjust it upwards for you in our next release. I will note that 60s is a fairly long time, so I would recommend confirming that everything is performing as expected from your origin DB before requesting this. Happy to do it, though, just let me know and please provide the Hyperdrive IDs for which you would like this increase.

letincho5•8/18/25, 1:56 PM

Ok, thanks for the answer. I guess that the Neon inactivity after 5 minutes... may affect too, no? The Neon wake-up times, may throw this error too?

Lletincho5 Ok, thanks for the answer. I guess that the Neon inactivity after 5 minutes... m...

AJROP•8/18/25, 1:56 PM

If their cold starts delay responses past the 60 second mark, that could cause this, yes. I haven't personally witnessed that, but it's certainly possible in theory.

letincho5•8/18/25, 1:57 PM

Oki, this is my actual configuration:

letincho5•8/18/25, 1:58 PM

I've better results with this, than without Pool config... but, in threory, is neccesary?

import { createConnection, type Connection } from "mysql2/promise"; const registry = new FinalizationRegistry((value) => { console.log(`"${value}" was GC'd`); }); console.log("FinalizationRegistry created"); const connections: WeakRef<Connection>[] = []; export default defineEventHandler(async (event) => { const { env } = event.context.cloudflare; const connection = await createConnection({ host: env.HYPERDRIVE.host, user: env.HYPERDRIVE.user, password: env.HYPERDRIVE.password, database: env.HYPERDRIVE.database, port: env.HYPERDRIVE.port, disableEval: true }); try { registry.register(connection, "MySQL2 connection"); connections.push(new WeakRef(connection)); await connection.query("SELECT * FROM `my_table`;"); event.waitUntil(connection.end()); return Response.json({ connections: connections.filter((ref) => ref.deref() !== undefined).length }); } catch (e) { console.error(e); } });

You know what, that's perfectly reasonable. Let me see if I can knock this one out for our next rele

Similar Threads

Similar Threads

Similar Threads