Serverless Driver for Java
Hey!
Has their been any plan to implement the Neon Serverless Driver for Java?
Creating a AWS Lambda Java Function and the normal JDBC connection takes 2 seconds for a mere initial connection. Looking for a way to get this down
Thanks
33 Replies
adverse-sapphire•2y ago
It'd be awesome to see if we could make improvements in this space, but I'm not aware of anything on our short term roadmap. I wonder if using something like Quarkus and GraalVM could improve the startup time? Or is it purely the JDBC layer that is slow?
rival-black•2y ago
The serverless driver is just a small wrapper around the HTTP API, right? As a stop-gap solution, you could also just use the HTTP API.
other-emeraldOP•2y ago
Purely JDBC
I knew it existed at some point, but I couldn't find in the documentation for the life of me
Would you mind linking the documentation to that?
rival-black•2y ago
Let me poke the team because I can't find it either
other-emeraldOP•2y ago
👍 Would appreciate that
sunny-green•2y ago
Sorry, documentation is still in the works. Give me a few minutes and I'll give you some pointers ...
The basic idea is that you https POST some JSON to the
/sql endpoint of your DB host, with a Neon-Connection-String header:
This returns:
There are two optional request headers to be aware of:
* Neon-Array-Mode: true makes the rows object an array of arrays instead of an array of objects (e.g. the above becomes [["hello world"]] instead of [{"greeting":"hello world"}]). This saves some bandwidth, and you can still retrieve the column names from the fields field.
* Neon-Raw-Text-Output: true disables parsing Postgres arrays into JSON arrays, so that the raw Postgres text format is always returned instead. For instance, try toggling Neon-Raw-Text-Output from true to false here:
Lastly, you can submit multiple queries in one transaction with one POST request, using the Neon-Batch-Isolation-Level header. For instance:
This returns:
Obviously you can use other isolation levels, and there are also headers Neon-Batch-Read-Only: true and Neon-Batch-Deferrable: true available to further configure the Postgres transaction.
Do note that at present the "params" key is required, and should be an empty array if no parameters need to be passed.
(Also, if you have jq installed, pipe all my curl examples into that for much more readable output).other-emeraldOP•2y ago
Appreciate it, thanks ❤️
How does this pair in with Prepared statement efficiency?
How does the response look with an
UPDATE ?
@Tristan Partin
Any idea?rival-black•2y ago
@Sword | Andrew let me ping internally
I pinged jawj internally. Note you might not hear from them until next week though
I haven't interacted with the HTTP API before. I think the best you can do if you don't want to wait is read the serverless driver code, which I know is not a great answer
sunny-green•2y ago
On prepared statements: you can't use prepared statements with the HTTPS transport. I think it's rare that this would be a problem unless you're running some truly enormous joins.
On an UPDATE response, I can send this:
and get back:
Or I could use RETURNING:
And get:
@Sword | Andrew Hope that helps.
other-emeraldOP•2y ago
That exactly what I needed, appreciate it ❤️
Is Binary Data served in a
BinaryNode compatible manner?
This isn't going to change without notice, correct? Just double checking as no documentation has been released
Are Datatype OIDs returned?
@jawj @Tristan Partin Apologies for the ping
The first field channel_id is a bigint in Neon. According to the JDBC Types Class, BIGINTshould have the Id of -5; Neon returns 20.
Is their anything I'm missing?rival-black•2y ago
Do you know where JDBC derives these values from
rival-black•2y ago
GitHub
neon/proxy/src/serverless/sql_over_http.rs at 7af4c676c01bd065fff6d...
Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, branching, and bottomless storage. - neondatabase/neon
rival-black•2y ago
investigating!
other-emeraldOP•2y ago
Let me find the exact class I pulled it from
Didn't you guys give
bigint Type Id 20?rival-black•2y ago
GitHub
pgjdbc/pgjdbc/src/main/java/org/postgresql/core/Oid.java at master ...
Postgresql JDBC Driver. Contribute to pgjdbc/pgjdbc development by creating an account on GitHub.
rival-black•2y ago
bigint is an alias for int8 in postgres
these values are the type's OID in postgres
rival-black•2y ago
GitHub
pgjdbc/pgjdbc/src/main/java/org/postgresql/jdbc/TypeInfoCache.java ...
Postgresql JDBC Driver. Contribute to pgjdbc/pgjdbc development by creating an account on GitHub.
rival-black•2y ago
Oid is defined as an unsigned int in postgres, so -5 would make no sense
typedef unsigned int Oid;
other-emeraldOP•2y ago
Maybe I'm missing something, from my understanding, Oid is an additional parameter returned with the Field
rival-black•2y ago
Oid is a numeric identifier for something in postgres, schemas, tables, indexes, types, etc
See the bit of rust code on how dataTypeId gets populated
other-emeraldOP•2y ago
I see, so this
dataTypeId is the Types OID; correct?rival-black•2y ago
Yes. We should probably rename the field
rival-black•2y ago
PostgreSQL Documentation
8.19. Object Identifier Types
8.19. Object Identifier Types # Object identifiers (OIDs) are used internally by PostgreSQL as primary keys for various system tables. Type …
other-emeraldOP•2y ago
JDBC has confused me it seems. It's Fields have a type OID and a dataTypeId. I assumed Neons return was the latter
rival-black•2y ago
I can see how that could be confusing
Would you want to open an issue in the Neon repo? There is good information here
other-emeraldOP•2y ago
@Tristan Partin
java.sql.Types is what I used as a reference
It's what JDBC uses as wellrival-black•2y ago
Here is where the java.sql.Types is translated to postgres OIDs https://github.com/pgjdbc/pgjdbc/blob/d91843a1c056ebe61343b1d3c0123bc42dcd7730/pgjdbc/src/main/java/org/postgresql/jdbc/PgPreparedStatement.java#L242-L244
GitHub
pgjdbc/pgjdbc/src/main/java/org/postgresql/jdbc/PgPreparedStatement...
Postgresql JDBC Driver. Contribute to pgjdbc/pgjdbc development by creating an account on GitHub.
other-emeraldOP•2y ago
Whoops, that was probably the one thing I shouldn't have skipped over 😅
Appreciate it
rival-black•2y ago
No problem! The beauty of open-souce is that we can read the code to find all this information
other-emeraldOP•2y ago
Postgres also has this array
sunny-green•2y ago
@Tristan Partin, thanks for digging into this. The result format we use here is intended to match what's returned by the standard JS pg library, node-postgres. That's because we may send some serverless driver
Pool.query queries over HTTP automatically in future, and we'd want that to be seamless from the developer perspective. So we're unlikely to change or rename anything that's returned here. We could probably consider adding things if there was a real use case for them, though.rival-black•2y ago
Thanks for the context!
other-emeraldOP•2y ago
@jawj
Are http requests made to PolyScale following this same format?