Even something like Postgres on the same server takes ~20-30ms to respond to a simple SELECT LIMIT 1

Even something like Postgres on the same server takes ~20-30ms to respond to a simple SELECT LIMIT 1 query