Cloudflare Developers•2y ago

hello anyone experiencing: Database error: error: Failed to acquire a connection from the pool.

hello anyone experiencing:
Database error: error: Failed to acquire a connection from the pool.

EElektra hello anyone experiencing: Database error: error: Failed to acquire a connection...

thomasgauvin•5/29/24, 4:09 PM

Any details on this? Using Hyperdrive, what's the database provider, etc.?

ElektraOP•5/29/24, 4:20 PM

Hi Thomas, I'm hosting my own instance of pg I'm using the recommended library for connecting. I never had any such issues before that's why I was asking. Been fine for the last 4 months since I've used it. I waited a bit then went to pgadmin to investigate (could have been my side, although I tested while waiting with other apis that I have apart from this worker and worked fine) and about 5 min after all was okay again. Went to the status page also of cloudflare but didn't see anything at the time. Maybe some transient error...

ElektraOP•5/29/24, 4:23 PM

By the the the connections was not exhausted at all. I have the stock 100 and only 7 were active. Although I now don't recall seeing cloudflare mentioned on the connections... So not sure... Unfortunately I do see the errors in the dashboard but... I can't see anything else except from that log message that I was getting. So I don't really have much more information at the moment.

berkinovish•5/31/24, 7:11 AM

While waiting for Cloudflare Tunnel support for Hyperdrive, are there any resources available about connecting to AWS RDS (via RDS Proxy) directly through a Cloudflare Tunnel from a worker?

ink pot monkey ✒🐒•5/31/24, 2:20 PM

Helloo everyone

I facing a puzzling situation. I have a postgres database that I am connecting to via hyperdrive that is run on a server with a firewall which has all of cloudflare's IP's whitelisted. When I trigger the cloudflare worker that uses hyperdrive to do database operations from my home (in Spain) this works. However this worker is a callback handler and whenever the service connects (from Slovakia) the database operations silently fail. I remember working with our sysAdmin to whitelist a different IP so my questions would be, is failure something to do with the firewall? What are the IP's that hyperdrive uses to connect to databases? Would these IP's be different between Spain and Slovakia. And if not does anyone have any idea what could be happening?

ramnivas•6/4/24, 2:18 AM

Is there a relationship between Hyperdrive cache settings and the use of prepared statements?

I am exploring Postgres through Rust (using the tokio-postgrestokio-postgres crate) using the

query

query

method. This uses the extended query protocol. I am observing the following behavior. When the Hyperdrive cache is:

Disabled: All (or most) queries fail (the driver reports an UnexpectedMessageUnexpectedMessage error). I suspect disabled caching makes Hyperdrive use one connection for preparation and another execution on another. Occasionally, it might use the same connection, leading to a successful query.
Enabled: All queries succeed. I suspect caching makes Hyperdrive choose the same connection it used to prepare the statement.

Am I interpreting this correctly? Is this the expected behavior?

p.s. This seems to have changed recently. Earlier,

query

query

would always fail regardless of the cache setting. So, I started working on a PR for the Postgres driver to make it work with Hyperdrive. But today, during testing that PR, I saw this behavior, thus this question.

Client in tokio_postgres - Rust

An asynchronous PostgreSQL client.

moody 💭•6/4/24, 5:16 AM

is something going on with cloudflare?

moody 💭•6/4/24, 5:17 AM

as of 2 hours ago my workers are performing terribly, same number of requests

Mitya•6/4/24, 10:01 AM

Two questions for the CF Hyperdrive team:

Given HD's caching capabilities, is it redundant to also utilise read replicas on my DB, or would they still be a good idea?
Six months or so ago I and others noticed that HD had a bug whereby, if caching was enabled, sometimes queries returned erroneous results e.g. no rows where there should definitely be rows. I recall vaguely that a CF dev said this was acknowledged and a fix was scheduled. Did this happen, i.e. is it safe for me to re-enable caching? (I disabled it at the time due to this).
Thanks in advance.

moody 💭•6/4/24, 3:11 PM

i seem to be having serious performance issues with hyperdrive connected to a postgres db in hetzner.
why is latency like this? other graphs are relatively flat, but errors and latency are spiking and users are reporting slowness.

Screenshot_2024-06-04_at_11.10.30_AM.png

Screenshot_2024-06-04_at_11.10.39_AM.png

Paul•6/4/24, 3:25 PM

Can we get some verbiage on the Supabase + Hyperdrive setup page in regards to the Supabase pooler? Do we use the pooler connection string or the direct connection string from the Supabase Database settings page?
https://developers.cloudflare.com/hyperdrive/examples/supabase/

Cloudflare Docs

Connect to Supabase · Hyperdrive

This example shows you how to connect Hyperdrive to a Supabase Postgres database.

Mmoody 💭i seem to be having serious performance issues with hyperdrive connected to a po...

moody 💭•6/4/24, 3:36 PM

all users reporting issues seem to be in asia

Paul•6/4/24, 4:27 PM

and so it begins... for some reason i am not able to see the HYPERDRIVE binding from getPlatformProxy within my SvelteKit hooks handler

AJR•6/4/24, 4:58 PM

Taking a look now.

MMitya Two questions for the CF Hyperdrive team: 1. Given HD's caching capabilities, is...

AJR•6/4/24, 5:00 PM

Depends on your use case. There's overlap in the problems they solve, certainly, but there's also reasons you might want both.
Yes, the issue with caching returning empty results was resolved quite a while back

Rramnivas Is there a relationship between Hyperdrive cache settings and the use of prepare...

AJR•6/4/24, 5:01 PM

We recently added support for prepared statements. Caching should not interact with it in this way, no. I'll add a ticket for us to take a look at this.

AAJR We recently added support for prepared statements. Caching should not interact w...

ramnivas•6/4/24, 6:26 PM

Thanks. That's good to know. I guess https://developers.cloudflare.com/hyperdrive/configuration/connect-to-postgres/#transaction-and-statement-support needs to change to drop the limitation on PREPARE?

Cloudflare Docs

Connect to PostgreSQL · Hyperdrive

Hyperdrive supports PostgreSQL and PostgreSQL-compatible databases, popular drivers and Object Relational Mapper (ORM) libraries that use those …

Rramnivas Thanks. That's good to know. I guess https://developers.cloudflare.com/hyperdriv...

AJR•6/4/24, 6:34 PM

Not exactly. We support protocol-level prepared statements, like your driver uses. We do not parse out and manage it as a keyword right now. At some point if there's enough need we could but we don't see that a lot honestly

ramnivas•6/4/24, 6:41 PM

Ah, I see. Thank you for the clarification.

AAJR Not exactly. We support protocol-level prepared statements, like your driver use...

ramnivas•6/4/24, 6:50 PM

BTW, a way to reproduce would be to take this example: https://github.com/cloudflare/workers-rs/blob/main/examples/tokio-postgres/src/lib.rs#L29 and change the line to use

query

query

instead of simple_querysimple_query.

GitHub

workers-rs/examples/tokio-postgres/src/lib.rs at main · cloudflare/...

Write Cloudflare Workers in 100% Rust via WebAssembly - cloudflare/workers-rs

Rramnivas BTW, a way to reproduce would be to take this example: https://github.com/cloudf...

Matt Silverlock•6/4/24, 7:13 PM

See https://github.com/sfackler/rust-postgres/pull/1067

GitHub

support unnamed statements by devsnek · Pull Request #1067 · sfackl...

Refs: #1017
Refs: #1044
This PR implements unnamed statements when querying the database. This is not a performance optimization. The library still uses a "prepared" statement in order to...

Matt Silverlock•6/4/24, 7:13 PM

@moody

(I can't DM you) - https://www.cloudflarestatus.com/incidents/q23jjvwggp78

for clarity, we only have one report so far, but are investigating.

Intermittent Hyperdrive connection issues in APJC

Rramnivas BTW, a way to reproduce would be to take this example: https://github.com/cloudf...

kian•6/4/24, 7:15 PM

Yeah, like Matt linked to, that example uses simple_querysimple_query intentionally.

https://github.com/cloudflare/workers-rs/pull/569#issuecomment-2098304851

AAJR We recently added support for prepared statements. Caching should not interact w...

ramnivas•6/4/24, 7:23 PM

@kian @elithrar @AJR Thanks for your responses.

Is there a document that describes the pooling mechanism? I am particularly interested in how prepared statements behave if I make two identical queries.

let res1 = client.query("SELECT * FROM FOO", &[]).await?;
let res2 = client.query("SELECT * FROM FOO", &[]).await?;

let res1 = client.query("SELECT * FROM FOO", &[]).await?;
let res2 = client.query("SELECT * FROM FOO", &[]).await?;

Would the statement prepared in the first query be cached? In other words, would Hyperdrive call prepareprepare twice (as in https://github.com/sfackler/rust-postgres/blob/98f5a11bc0a8e451552d8941ffa078c7eb6cd60c/tokio-postgres/src/to_statement.rs#L18, which is invoked from https://github.com/sfackler/rust-postgres/blob/98f5a11bc0a8e451552d8941ffa078c7eb6cd60c/tokio-postgres/src/client.rs#L363.

A bit of context:

I am in the process of supporting Cloudflare Worker for Exograph (https://github.com/exograph/exograph/pull/1081). It is working okay without Hyperdrive (high latency) and great with Hyperdrive (with a fork of tokio-postgres to avoid prepared statements: https://github.com/sfackler/rust-postgres/pull/1147).

I would like to determine the extent of Hyperdrive support to decide if there is value in continuing to use my fork. For example, if Hyperdrive reuses the prepared statement, then going with

query

query

will be better (to be measured, of course). If not, my fork avoids an extra roundtrip.

Rramnivas @kian @elithrar @AJR Thanks for your responses. Is there a document that descri...

AJR•6/4/24, 7:27 PM

I don't think this is well documented. I'm happy to write up some explanation when I have some spare cycles. Hopefully today or tomorrow.

Short answer, it's complicated. We reuse the prepared statement if possible, or we re-send the parse on the new connection if not.

MMatt Silverlock @moody 💭 (I can't DM you) - https://www.cloudflarestatus.com/incidents/q23jjvwg...

moody 💭•6/4/24, 7:27 PM

ty. i think the no dms setting is on your end

AAJR I don't think this is well documented. I'm happy to write up some explanation wh...

ramnivas•6/4/24, 7:29 PM

That will be awesome. Thank you!

Kkian Yeah, like Matt linked to, that example uses `simple_query` intentionally. <htt...

ramnivas•6/4/24, 10:15 PM

Just a clarification: The use of simple_querysimple_query was required earlier (I believe until May 22 https://developers.cloudflare.com/hyperdrive/platform/changelog/#2024-05-22). With the prepared statement support in Hyperdrive, plain

query

query

also works with a caveat I mentioned (must turn on caching in Hyperdrive setting). The caveat is an issue in Hyperdrive (per @AJR), so once that is fixed, it will be all okay!

Cloudflare Docs

Changelog · Hyperdrive

You can now create up to 25 Hyperdrive configurations per account, up from the previous maximum of 10.

Paul•6/5/24, 3:44 AM

Is Hyperdrive smart enough to clear the cache of a SELECT query if a mutate has happened for that row? For instance, the user profile is cached but then they change their bio. If they refresh would they still see the old profile until “max-age”?

PPaul Is Hyperdrive smart enough to clear the cache of a SELECT query if a mutate has ...

Isaac McFadyen•6/5/24, 4:47 AM

max-agemax-age is related to browser-side caching using HTTP headers. Hyperdrive has no ability to purge the browser-side cache so no.

PPaul Is Hyperdrive smart enough to clear the cache of a SELECT query if a mutate has ...

AJR•6/5/24, 11:58 AM

Without getting into browser caching, no Hyperdrive does not do proactive cache purging after volatile queries. It's something we've discussed but would be a non-trivial feature to add and generally is not something users have asked for overmuch.

If you need to bypass cache on specific queries, we don't check cache on any volatile queries, so you could add any such function. For example including random()random() and ignoring that field in your result would do it.

If not purging cache on mutation is a dealbreaker feature for you, it would be helpful for us to understand a bit about your use case, if you'd be willing to chat about it.

AAJR Without getting into browser caching, no Hyperdrive does not do proactive cache ...

Isaac McFadyen•6/5/24, 1:24 PM

I'd also add that max-age is also browser-side along with server-side.

Isaac McFadyen•6/5/24, 1:24 PM

So even if Hyperdrive could purge the cache somehow (which seems like a bad idea IMO, a DB connection shouldn't be able to modify my CDN config) then it won't affect browser-side caching anyway.

Paul•6/5/24, 2:32 PM

Hmm seems to be a few caching layers here. Let’s say the user who owns the profile has edited it and expects to see the changes immediately. Is there a way to cache bust just for that user? (I’m not understanding how the browser cache plays along with the hyperdrive layer)

PPaul Hmm seems to be a few caching layers here. Let’s say the user who owns the profi...

Isaac McFadyen•6/5/24, 2:43 PM

So these are the cache layers I assume you have (correct me here if I'm wrong):

Browser cache, this is local on the user's web browser and based on max-agemax-age. This is quite tricky to bust and you usually disable this for APIs in favor of 2.
Server-side cache, this is the Cloudflare CDN and/or your server and/or the Worker that is serving the API requests. To purge this you need control of the Worker, or if it's the CDN cache you need to make an API call to the Cloudflare API. This is also based on maxagemaxage, but if you want to only cache on the server-side you can use s-maxages-maxage instead.
Hyperdrive query cache. I don't know many details on this, @AJR would know more about how cache busting works here.

Isaac McFadyen•6/5/24, 2:44 PM

Your original question was about whether a query in Hyperdrive would auto-cache-purge 1 or 2. It doesn't do that because it's a separate system and it's unintuitive/broken behavior if a database connection cache purges the other layers. The developer should have full control over whether that happens and if so, when and how.

AJR•6/5/24, 2:59 PM

That is all correct to the best of my understanding. My answer above about including a benign volatile function is the current best solution for cache busting point 3. We expect to add more polish on that front, though it isn't our highest priority at this time.

AAJR That is all correct to the best of my understanding. My answer above about inclu...

Isaac McFadyen•6/5/24, 3:39 PM

Ah completely missed the volatile function part - good to know.

IIsaac McFadyen So these are the cache layers I assume you have (correct me here if I'm wrong): ...

Paul•6/5/24, 4:06 PM

Thanks for the great write up!

IIsaac McFadyen Your original question was about whether a query in Hyperdrive would auto-cache-...

Matt Silverlock•6/5/24, 10:04 PM

I do think some things are being mixed up here:

Hyperdrive’s cache is only specific to Hyperdrive (per config ID). The maxagemaxage in Hyperdrive dictates that caches maximum TTL per unique query. It’s not a CDN cache and doesn’t cache any results in the browser. maxagemaxage is just a common name for a TTL-based cache configuration setting.
You can also use Cloudflare’s CDN cache from a Worker or Pages app, but there is no intersection with Hyperdrive (unless you actively choose to insert things into it).

MMatt Silverlock I do think some things are being mixed up here: 1. Hyperdrive’s cache is only s...

Isaac McFadyen•6/5/24, 10:05 PM

Ah, think you're right - apologies, thought this was referring to the maxagemaxage in the Cache-ControlCache-Control header.

Muudka•6/5/24, 11:48 PM

Hi, am a bit confused about hyperdrive, at the moment i have a node application that is deployed to aws lamda behind api gateway ,
I am using postgres db with prisma orm,
I have set up hyperdrive and run some test, how can I integrate it into my lambda deployment

MMuudka Hi, am a bit confused about hyperdrive, at the moment i have a node applicati...

Matt Silverlock•6/6/24, 1:57 AM

Where is your database? Hyperdrive is currently focused on making it faster to connect (and query) your database from Workers.

If you’re using Lambda and Aurora in the same region, it won’t make much sense here.

ramnivas•6/6/24, 2:01 AM

I have seen mentions in this/workers forum that Hyperdrive's

id

id

is not a secret, but I didn't find that mentioned in the documentation. It will be beneficial to have that mentioned officially.

MMatt Silverlock Where is your database? Hyperdrive is currently focused on making it faster to c...

Muudka•6/6/24, 6:34 AM

I am using postgtres on Neo.tech, does worker only work within the worker and how can it be incorporated into my code.
I am using prisma as ORM

MMuudka I am using postgtres on Neo.tech, does worker only work within the worker and ho...

thomasgauvin•6/6/24, 3:23 PM

@Muudka are you using AWS Lambda or Cloudflare Workers (I see mentions of both in this thread)? Which database vendor are you using/is it AWS RDS/Aurora?

hello anyone experiencing: Database error: error: Failed to acquire a connection from the pool.

Similar Threads

Similar Threads

Similar Threads