N
Neon2y ago
other-emerald

Serverless Driver for Java

Hey! Has their been any plan to implement the Neon Serverless Driver for Java? Creating a AWS Lambda Java Function and the normal JDBC connection takes 2 seconds for a mere initial connection. Looking for a way to get this down Thanks
33 Replies
adverse-sapphire
adverse-sapphire2y ago
It'd be awesome to see if we could make improvements in this space, but I'm not aware of anything on our short term roadmap. I wonder if using something like Quarkus and GraalVM could improve the startup time? Or is it purely the JDBC layer that is slow?
rival-black
rival-black2y ago
The serverless driver is just a small wrapper around the HTTP API, right? As a stop-gap solution, you could also just use the HTTP API.
other-emerald
other-emeraldOP2y ago
Purely JDBC I knew it existed at some point, but I couldn't find in the documentation for the life of me Would you mind linking the documentation to that?
rival-black
rival-black2y ago
Let me poke the team because I can't find it either
other-emerald
other-emeraldOP2y ago
👍 Would appreciate that
sunny-green
sunny-green2y ago
Sorry, documentation is still in the works. Give me a few minutes and I'll give you some pointers ... The basic idea is that you https POST some JSON to the /sql endpoint of your DB host, with a Neon-Connection-String header:
curl -v 'https://ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/sql' \
-H 'Neon-Connection-String: postgres://user:password@ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/main' \
-H 'Content-Type: application/json' \
--data '{
"query": "SELECT $1 AS greeting",
"params": [ "hello world" ]
}'
curl -v 'https://ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/sql' \
-H 'Neon-Connection-String: postgres://user:password@ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/main' \
-H 'Content-Type: application/json' \
--data '{
"query": "SELECT $1 AS greeting",
"params": [ "hello world" ]
}'
This returns:
{
"command": "SELECT",
"fields": [
{
"columnID": 0,
"dataTypeID": 25,
"dataTypeModifier": -1,
"dataTypeSize": -1,
"format": "text",
"name": "greeting",
"tableID": 0
}
],
"rowAsArray": false,
"rowCount": 1,
"rows": [
{
"greeting": "hello world"
}
]
}
{
"command": "SELECT",
"fields": [
{
"columnID": 0,
"dataTypeID": 25,
"dataTypeModifier": -1,
"dataTypeSize": -1,
"format": "text",
"name": "greeting",
"tableID": 0
}
],
"rowAsArray": false,
"rowCount": 1,
"rows": [
{
"greeting": "hello world"
}
]
}
There are two optional request headers to be aware of: * Neon-Array-Mode: true makes the rows object an array of arrays instead of an array of objects (e.g. the above becomes [["hello world"]] instead of [{"greeting":"hello world"}]). This saves some bandwidth, and you can still retrieve the column names from the fields field. * Neon-Raw-Text-Output: true disables parsing Postgres arrays into JSON arrays, so that the raw Postgres text format is always returned instead. For instance, try toggling Neon-Raw-Text-Output from true to false here:
curl -v 'https://ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/sql' \
-H 'Neon-Connection-String: postgres://user:password@ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/main' \
-H 'Content-Type: application/json' \
-H 'Neon-Raw-Text-Output: true' \
--data '{
"query": "SELECT $1::int[] AS pgarray",
"params": [ [1, 2, 3] ]
}'
curl -v 'https://ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/sql' \
-H 'Neon-Connection-String: postgres://user:password@ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/main' \
-H 'Content-Type: application/json' \
-H 'Neon-Raw-Text-Output: true' \
--data '{
"query": "SELECT $1::int[] AS pgarray",
"params": [ [1, 2, 3] ]
}'
Lastly, you can submit multiple queries in one transaction with one POST request, using the Neon-Batch-Isolation-Level header. For instance:
curl -v 'https://ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/sql' \
-H 'Neon-Connection-String: postgres://user:password@ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/main' \
-H 'Content-Type: application/json' \
-H 'Neon-Batch-Isolation-Level: Serializable' \
--data '{
"queries": [
{ "query": "SELECT now() AS today", "params": [] },
{ "query": "SELECT now() + $1::interval AS tomorrow", "params": ["1 day"] }
]
}'
curl -v 'https://ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/sql' \
-H 'Neon-Connection-String: postgres://user:password@ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/main' \
-H 'Content-Type: application/json' \
-H 'Neon-Batch-Isolation-Level: Serializable' \
--data '{
"queries": [
{ "query": "SELECT now() AS today", "params": [] },
{ "query": "SELECT now() + $1::interval AS tomorrow", "params": ["1 day"] }
]
}'
This returns:
{
"results": [
{
"command": "SELECT",
"fields": [
{
"columnID": 0,
"dataTypeID": 1184,
"dataTypeModifier": -1,
"dataTypeSize": 8,
"format": "text",
"name": "today",
"tableID": 0
}
],
"rowAsArray": false,
"rowCount": 1,
"rows": [
{
"today": "2024-01-09 10:00:41.670368+00"
}
]
},
{
"command": "SELECT",
"fields": [
{
"columnID": 0,
"dataTypeID": 1184,
"dataTypeModifier": -1,
"dataTypeSize": 8,
"format": "text",
"name": "tomorrow",
"tableID": 0
}
],
"rowAsArray": false,
"rowCount": 1,
"rows": [
{
"tomorrow": "2024-01-10 10:00:41.670368+00"
}
]
}
]
}
{
"results": [
{
"command": "SELECT",
"fields": [
{
"columnID": 0,
"dataTypeID": 1184,
"dataTypeModifier": -1,
"dataTypeSize": 8,
"format": "text",
"name": "today",
"tableID": 0
}
],
"rowAsArray": false,
"rowCount": 1,
"rows": [
{
"today": "2024-01-09 10:00:41.670368+00"
}
]
},
{
"command": "SELECT",
"fields": [
{
"columnID": 0,
"dataTypeID": 1184,
"dataTypeModifier": -1,
"dataTypeSize": 8,
"format": "text",
"name": "tomorrow",
"tableID": 0
}
],
"rowAsArray": false,
"rowCount": 1,
"rows": [
{
"tomorrow": "2024-01-10 10:00:41.670368+00"
}
]
}
]
}
Obviously you can use other isolation levels, and there are also headers Neon-Batch-Read-Only: true and Neon-Batch-Deferrable: true available to further configure the Postgres transaction. Do note that at present the "params" key is required, and should be an empty array if no parameters need to be passed. (Also, if you have jq installed, pipe all my curl examples into that for much more readable output).
other-emerald
other-emeraldOP2y ago
Appreciate it, thanks ❤️ How does this pair in with Prepared statement efficiency? How does the response look with an UPDATE ? @Tristan Partin Any idea?
rival-black
rival-black2y ago
@Sword | Andrew let me ping internally I pinged jawj internally. Note you might not hear from them until next week though I haven't interacted with the HTTP API before. I think the best you can do if you don't want to wait is read the serverless driver code, which I know is not a great answer
sunny-green
sunny-green2y ago
On prepared statements: you can't use prepared statements with the HTTPS transport. I think it's rare that this would be a problem unless you're running some truly enormous joins. On an UPDATE response, I can send this:
curl -v 'https://ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/sql' \
-H 'Neon-Connection-String: postgres://user:password@ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/main' \
-H 'Content-Type: application/json' \
--data '{
"query": "UPDATE test SET name = $1 WHERE id = $2",
"params": ["X", 1]
}'
curl -v 'https://ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/sql' \
-H 'Neon-Connection-String: postgres://user:password@ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/main' \
-H 'Content-Type: application/json' \
--data '{
"query": "UPDATE test SET name = $1 WHERE id = $2",
"params": ["X", 1]
}'
and get back:
{
"command": "UPDATE",
"fields": [],
"rowAsArray": false,
"rowCount": 1,
"rows": []
}
{
"command": "UPDATE",
"fields": [],
"rowAsArray": false,
"rowCount": 1,
"rows": []
}
Or I could use RETURNING:
curl -v 'https://ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/sql' \
-H 'Neon-Connection-String: postgres://user:password@ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/main' \
-H 'Content-Type: application/json' \
--data '{
"query": "UPDATE test SET name = $1 WHERE id = $2 RETURNING *",
"params": ["X", 1]
}'
curl -v 'https://ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/sql' \
-H 'Neon-Connection-String: postgres://user:password@ep-adjective-noun-nnnn.aws-region-n.aws.neon.tech/main' \
-H 'Content-Type: application/json' \
--data '{
"query": "UPDATE test SET name = $1 WHERE id = $2 RETURNING *",
"params": ["X", 1]
}'
And get:
{
"command": "UPDATE",
"fields": [
{
"columnID": 1,
"dataTypeID": 23,
"dataTypeModifier": -1,
"dataTypeSize": 4,
"format": "text",
"name": "id",
"tableID": 122888
},
{
"columnID": 2,
"dataTypeID": 25,
"dataTypeModifier": -1,
"dataTypeSize": -1,
"format": "text",
"name": "name",
"tableID": 122888
}
],
"rowAsArray": false,
"rowCount": 1,
"rows": [
{
"id": 1,
"name": "X"
}
]
}
{
"command": "UPDATE",
"fields": [
{
"columnID": 1,
"dataTypeID": 23,
"dataTypeModifier": -1,
"dataTypeSize": 4,
"format": "text",
"name": "id",
"tableID": 122888
},
{
"columnID": 2,
"dataTypeID": 25,
"dataTypeModifier": -1,
"dataTypeSize": -1,
"format": "text",
"name": "name",
"tableID": 122888
}
],
"rowAsArray": false,
"rowCount": 1,
"rows": [
{
"id": 1,
"name": "X"
}
]
}
@Sword | Andrew Hope that helps.
other-emerald
other-emeraldOP2y ago
That exactly what I needed, appreciate it ❤️ Is Binary Data served in a BinaryNode compatible manner? This isn't going to change without notice, correct? Just double checking as no documentation has been released Are Datatype OIDs returned? @jawj @Tristan Partin Apologies for the ping
{
"command": "SELECT",
"fields": [{
"columnID": 2,
"dataTypeID": 20,
"dataTypeModifier": -1,
"dataTypeSize": 8,
"format": "text",
"name": "channel_id",
"tableID": 148344
}, {
"columnID": 5,
"dataTypeID": 2950,
"dataTypeModifier": -1,
"dataTypeSize": 16,
"format": "text",
"name": "category_id",
"tableID": 147932
}, {
"columnID": 6,
"dataTypeID": 25,
"dataTypeModifier": -1,
"dataTypeSize": -1,
"format": "text",
"name": "display_name",
"tableID": 147932
}, {
"columnID": 7,
"dataTypeID": 25,
"dataTypeModifier": -1,
"dataTypeSize": -1,
"format": "text",
"name": "topic",
"tableID": 147932
}
],
"rowAsArray": true,
"rowCount": 1,
"rows": [["<redacted>", null, "<redacted>", "<redacted>"]]
}
{
"command": "SELECT",
"fields": [{
"columnID": 2,
"dataTypeID": 20,
"dataTypeModifier": -1,
"dataTypeSize": 8,
"format": "text",
"name": "channel_id",
"tableID": 148344
}, {
"columnID": 5,
"dataTypeID": 2950,
"dataTypeModifier": -1,
"dataTypeSize": 16,
"format": "text",
"name": "category_id",
"tableID": 147932
}, {
"columnID": 6,
"dataTypeID": 25,
"dataTypeModifier": -1,
"dataTypeSize": -1,
"format": "text",
"name": "display_name",
"tableID": 147932
}, {
"columnID": 7,
"dataTypeID": 25,
"dataTypeModifier": -1,
"dataTypeSize": -1,
"format": "text",
"name": "topic",
"tableID": 147932
}
],
"rowAsArray": true,
"rowCount": 1,
"rows": [["<redacted>", null, "<redacted>", "<redacted>"]]
}
The first field channel_id is a bigint in Neon. According to the JDBC Types Class, BIGINTshould have the Id of -5; Neon returns 20. Is their anything I'm missing?
rival-black
rival-black2y ago
Do you know where JDBC derives these values from
rival-black
rival-black2y ago
GitHub
neon/proxy/src/serverless/sql_over_http.rs at 7af4c676c01bd065fff6d...
Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, branching, and bottomless storage. - neondatabase/neon
rival-black
rival-black2y ago
investigating!
select * from pg_type where typname = 'int8';
-[ RECORD 1 ]--+---------
oid | 20
typname | int8
typnamespace | 11
typowner | 10
typlen | 8
typbyval | t
typtype | b
typcategory | N
typispreferred | f
typisdefined | t
typdelim | ,
typrelid | 0
typsubscript | -
typelem | 0
typarray | 1016
typinput | int8in
typoutput | int8out
typreceive | int8recv
typsend | int8send
typmodin | -
typmodout | -
typanalyze | -
typalign | d
typstorage | p
typnotnull | f
typbasetype | 0
typtypmod | -1
typndims | 0
typcollation | 0
typdefaultbin | (null)
typdefault | (null)
typacl | (null)
select * from pg_type where typname = 'int8';
-[ RECORD 1 ]--+---------
oid | 20
typname | int8
typnamespace | 11
typowner | 10
typlen | 8
typbyval | t
typtype | b
typcategory | N
typispreferred | f
typisdefined | t
typdelim | ,
typrelid | 0
typsubscript | -
typelem | 0
typarray | 1016
typinput | int8in
typoutput | int8out
typreceive | int8recv
typsend | int8send
typmodin | -
typmodout | -
typanalyze | -
typalign | d
typstorage | p
typnotnull | f
typbasetype | 0
typtypmod | -1
typndims | 0
typcollation | 0
typdefaultbin | (null)
typdefault | (null)
typacl | (null)
other-emerald
other-emeraldOP2y ago
Let me find the exact class I pulled it from Didn't you guys give bigint Type Id 20?
rival-black
rival-black2y ago
rival-black
rival-black2y ago
bigint is an alias for int8 in postgres these values are the type's OID in postgres
rival-black
rival-black2y ago
Oid is defined as an unsigned int in postgres, so -5 would make no sense typedef unsigned int Oid;
other-emerald
other-emeraldOP2y ago
Maybe I'm missing something, from my understanding, Oid is an additional parameter returned with the Field
rival-black
rival-black2y ago
Oid is a numeric identifier for something in postgres, schemas, tables, indexes, types, etc See the bit of rust code on how dataTypeId gets populated
other-emerald
other-emeraldOP2y ago
I see, so this dataTypeId is the Types OID; correct?
rival-black
rival-black2y ago
Yes. We should probably rename the field
rival-black
rival-black2y ago
PostgreSQL Documentation
8.19. Object Identifier Types
8.19. Object Identifier Types # Object identifiers (OIDs) are used internally by PostgreSQL as primary keys for various system tables. Type …
other-emerald
other-emeraldOP2y ago
JDBC has confused me it seems. It's Fields have a type OID and a dataTypeId. I assumed Neons return was the latter
rival-black
rival-black2y ago
I can see how that could be confusing Would you want to open an issue in the Neon repo? There is good information here
other-emerald
other-emeraldOP2y ago
@Tristan Partin java.sql.Types is what I used as a reference It's what JDBC uses as well
rival-black
rival-black2y ago
GitHub
pgjdbc/pgjdbc/src/main/java/org/postgresql/jdbc/PgPreparedStatement...
Postgresql JDBC Driver. Contribute to pgjdbc/pgjdbc development by creating an account on GitHub.
other-emerald
other-emeraldOP2y ago
Whoops, that was probably the one thing I shouldn't have skipped over 😅 Appreciate it
rival-black
rival-black2y ago
No problem! The beauty of open-souce is that we can read the code to find all this information
other-emerald
other-emeraldOP2y ago
Postgres also has this array
private static final Object[][] types = {
{"int2", Oid.INT2, Types.SMALLINT, "java.lang.Integer", Oid.INT2_ARRAY},
{"int4", Oid.INT4, Types.INTEGER, "java.lang.Integer", Oid.INT4_ARRAY},
{"oid", Oid.OID, Types.BIGINT, "java.lang.Long", Oid.OID_ARRAY},
{"int8", Oid.INT8, Types.BIGINT, "java.lang.Long", Oid.INT8_ARRAY},
{"money", Oid.MONEY, Types.DOUBLE, "java.lang.Double", Oid.MONEY_ARRAY},
{"numeric", Oid.NUMERIC, Types.NUMERIC, "java.math.BigDecimal", Oid.NUMERIC_ARRAY},
{"float4", Oid.FLOAT4, Types.REAL, "java.lang.Float", Oid.FLOAT4_ARRAY},
{"float8", Oid.FLOAT8, Types.DOUBLE, "java.lang.Double", Oid.FLOAT8_ARRAY},
{"char", Oid.CHAR, Types.CHAR, "java.lang.String", Oid.CHAR_ARRAY},
{"bpchar", Oid.BPCHAR, Types.CHAR, "java.lang.String", Oid.BPCHAR_ARRAY},
{"varchar", Oid.VARCHAR, Types.VARCHAR, "java.lang.String", Oid.VARCHAR_ARRAY},
{"varbit", Oid.VARBIT, Types.OTHER, "java.lang.String", Oid.VARBIT_ARRAY},
{"text", Oid.TEXT, Types.VARCHAR, "java.lang.String", Oid.TEXT_ARRAY},
{"name", Oid.NAME, Types.VARCHAR, "java.lang.String", Oid.NAME_ARRAY},
{"bytea", Oid.BYTEA, Types.BINARY, "[B", Oid.BYTEA_ARRAY},
{"bool", Oid.BOOL, Types.BIT, "java.lang.Boolean", Oid.BOOL_ARRAY},
{"bit", Oid.BIT, Types.BIT, "java.lang.Boolean", Oid.BIT_ARRAY},
{"date", Oid.DATE, Types.DATE, "java.sql.Date", Oid.DATE_ARRAY},
{"time", Oid.TIME, Types.TIME, "java.sql.Time", Oid.TIME_ARRAY},
{"timetz", Oid.TIMETZ, Types.TIME, "java.sql.Time", Oid.TIMETZ_ARRAY},
{"timestamp", Oid.TIMESTAMP, Types.TIMESTAMP, "java.sql.Timestamp", Oid.TIMESTAMP_ARRAY},
{"timestamptz", Oid.TIMESTAMPTZ, Types.TIMESTAMP, "java.sql.Timestamp",
Oid.TIMESTAMPTZ_ARRAY},
{"refcursor", Oid.REF_CURSOR, Types.REF_CURSOR, "java.sql.ResultSet", Oid.REF_CURSOR_ARRAY},
{"json", Oid.JSON, Types.OTHER, "org.postgresql.util.PGobject", Oid.JSON_ARRAY},
{"point", Oid.POINT, Types.OTHER, "org.postgresql.geometric.PGpoint", Oid.POINT_ARRAY},
{"box", Oid.BOX, Types.OTHER, "org.postgresql.geometric.PGBox", Oid.BOX_ARRAY}
};
private static final Object[][] types = {
{"int2", Oid.INT2, Types.SMALLINT, "java.lang.Integer", Oid.INT2_ARRAY},
{"int4", Oid.INT4, Types.INTEGER, "java.lang.Integer", Oid.INT4_ARRAY},
{"oid", Oid.OID, Types.BIGINT, "java.lang.Long", Oid.OID_ARRAY},
{"int8", Oid.INT8, Types.BIGINT, "java.lang.Long", Oid.INT8_ARRAY},
{"money", Oid.MONEY, Types.DOUBLE, "java.lang.Double", Oid.MONEY_ARRAY},
{"numeric", Oid.NUMERIC, Types.NUMERIC, "java.math.BigDecimal", Oid.NUMERIC_ARRAY},
{"float4", Oid.FLOAT4, Types.REAL, "java.lang.Float", Oid.FLOAT4_ARRAY},
{"float8", Oid.FLOAT8, Types.DOUBLE, "java.lang.Double", Oid.FLOAT8_ARRAY},
{"char", Oid.CHAR, Types.CHAR, "java.lang.String", Oid.CHAR_ARRAY},
{"bpchar", Oid.BPCHAR, Types.CHAR, "java.lang.String", Oid.BPCHAR_ARRAY},
{"varchar", Oid.VARCHAR, Types.VARCHAR, "java.lang.String", Oid.VARCHAR_ARRAY},
{"varbit", Oid.VARBIT, Types.OTHER, "java.lang.String", Oid.VARBIT_ARRAY},
{"text", Oid.TEXT, Types.VARCHAR, "java.lang.String", Oid.TEXT_ARRAY},
{"name", Oid.NAME, Types.VARCHAR, "java.lang.String", Oid.NAME_ARRAY},
{"bytea", Oid.BYTEA, Types.BINARY, "[B", Oid.BYTEA_ARRAY},
{"bool", Oid.BOOL, Types.BIT, "java.lang.Boolean", Oid.BOOL_ARRAY},
{"bit", Oid.BIT, Types.BIT, "java.lang.Boolean", Oid.BIT_ARRAY},
{"date", Oid.DATE, Types.DATE, "java.sql.Date", Oid.DATE_ARRAY},
{"time", Oid.TIME, Types.TIME, "java.sql.Time", Oid.TIME_ARRAY},
{"timetz", Oid.TIMETZ, Types.TIME, "java.sql.Time", Oid.TIMETZ_ARRAY},
{"timestamp", Oid.TIMESTAMP, Types.TIMESTAMP, "java.sql.Timestamp", Oid.TIMESTAMP_ARRAY},
{"timestamptz", Oid.TIMESTAMPTZ, Types.TIMESTAMP, "java.sql.Timestamp",
Oid.TIMESTAMPTZ_ARRAY},
{"refcursor", Oid.REF_CURSOR, Types.REF_CURSOR, "java.sql.ResultSet", Oid.REF_CURSOR_ARRAY},
{"json", Oid.JSON, Types.OTHER, "org.postgresql.util.PGobject", Oid.JSON_ARRAY},
{"point", Oid.POINT, Types.OTHER, "org.postgresql.geometric.PGpoint", Oid.POINT_ARRAY},
{"box", Oid.BOX, Types.OTHER, "org.postgresql.geometric.PGBox", Oid.BOX_ARRAY}
};
sunny-green
sunny-green2y ago
@Tristan Partin, thanks for digging into this. The result format we use here is intended to match what's returned by the standard JS pg library, node-postgres. That's because we may send some serverless driver Pool.query queries over HTTP automatically in future, and we'd want that to be seamless from the developer perspective. So we're unlikely to change or rename anything that's returned here. We could probably consider adding things if there was a real use case for them, though.
rival-black
rival-black2y ago
Thanks for the context!
other-emerald
other-emeraldOP2y ago
@jawj Are http requests made to PolyScale following this same format?

Did you find this page helpful?