9 replies

Neptune Local Development Access Methods

Question for the Neptune folks - I'm trying to cobble together ergonomic local access to our development cluster.

Here's what I've got that works:
- SSM session running via an EC2 bastion host, which makes the targeted cluster (using

AWS-StartPortForwardingSessionToRemoteHost

AWS-StartPortForwardingSessionToRemoteHost

) available at

localhost:8182

localhost:8182

- A mapping in

/etc/hosts

/etc/hosts

resolving

<my_cluster_url>

<my_cluster_url>

localhost

localhost

- In combination, these enable me to use the same SigV4Auth signing approach (with headers) that works from a within-VPC context to access the cluster from a Python script/Jupyter notebook
- https://docs.aws.amazon.com/neptune/latest/userguide/access-graph-gremlin-python.html

I have been trying to come up with an approach for handling the

/etc/hosts

/etc/hosts

edit that doesn't require

sudo

sudo

access or changes to system files. In principle, it should be possible to insert custom DNS resolution into the Python script, but I'm having no joy. Has anyone else previously hit and overcome this snag in this fashion/another way?

Solution

Ok, after a bit more tinkering, here's an implementation that doesn't require any outside-script configuration or tools, and doesn't require disabling SSL.

It requires an additional import

AiohttpTransport

AiohttpTransport

which is used to pass the server hostname to

DriverRemoteConnection

DriverRemoteConnection

via

transport_factory

transport_factory

, solving the hostname mismatch failure issue that arises without it by passing the 'correct' host to the TLS handshake.

import boto3
from botocore.auth import SigV4Auth, _host_from_url
from botocore.awsrequest import AWSRequest
from botocore.credentials import Credentials
from gremlin_python.driver.aiohttp.transport import AiohttpTransport
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
from gremlin_python.process.anonymous_traversal import traversal

# # If Jupyter notebook
# import nest_asyncio
# nest_asyncio.apply()


def get_signed_headers(url: str, region: str, creds: Credentials) -> dict:
    request = AWSRequest(method="GET", url=url)
    request.headers.add_header("Host", _host_from_url(str(request.url)))
    SigV4Auth(creds, "neptune-db", region).add_auth(request)
    return dict(request.headers)


neptune_hostname = "mycluster.eu-west-2.neptune.amazonaws.com"
true_database_url = f"wss://{neptune_hostname}:8182/gremlin"
local_database_url = "wss://localhost:8182/gremlin"

creds = boto3.Session(profile_name="my_profile").get_credentials()
headers = get_signed_headers(true_database_url, "eu-west-2", creds)

conn = None

try:
    conn = DriverRemoteConnection(
        local_database_url,
        "g",
        headers=headers,
        transport_factory=lambda: AiohttpTransport(server_hostname=neptune_hostname),
    )

    g = traversal().withRemote(conn)

    # Return 1 if connection successful
    print(g.inject(1).toList())
finally:
    if conn:
        conn.close()

import boto3
from botocore.auth import SigV4Auth, _host_from_url
from botocore.awsrequest import AWSRequest
from botocore.credentials import Credentials
from gremlin_python.driver.aiohttp.transport import AiohttpTransport
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
from gremlin_python.process.anonymous_traversal import traversal

# # If Jupyter notebook
# import nest_asyncio
# nest_asyncio.apply()


def get_signed_headers(url: str, region: str, creds: Credentials) -> dict:
    request = AWSRequest(method="GET", url=url)
    request.headers.add_header("Host", _host_from_url(str(request.url)))
    SigV4Auth(creds, "neptune-db", region).add_auth(request)
    return dict(request.headers)


neptune_hostname = "mycluster.eu-west-2.neptune.amazonaws.com"
true_database_url = f"wss://{neptune_hostname}:8182/gremlin"
local_database_url = "wss://localhost:8182/gremlin"

creds = boto3.Session(profile_name="my_profile").get_credentials()
headers = get_signed_headers(true_database_url, "eu-west-2", creds)

conn = None

try:
    conn = DriverRemoteConnection(
        local_database_url,
        "g",
        headers=headers,
        transport_factory=lambda: AiohttpTransport(server_hostname=neptune_hostname),
    )

    g = traversal().withRemote(conn)

    # Return 1 if connection successful
    print(g.inject(1).toList())
finally:
    if conn:
        conn.close()

Jump to solution

Apache TinkerPop•10mo ago•

9 replies

Captator

Neptune Local Development Access Methods

AWS-StartPortForwardingSessionToRemoteHost

AWS-StartPortForwardingSessionToRemoteHost

) available at

localhost:8182

localhost:8182

- A mapping in

/etc/hosts

/etc/hosts

resolving

<my_cluster_url>

<my_cluster_url>

localhost

localhost

/etc/hosts

/etc/hosts

edit that doesn't require

sudo

sudo

Solution

Ok, after a bit more tinkering, here's an implementation that doesn't require any outside-script configuration or tools, and doesn't require disabling SSL.

It requires an additional import

AiohttpTransport

AiohttpTransport

which is used to pass the server hostname to

DriverRemoteConnection

DriverRemoteConnection

via

transport_factory

transport_factory

, solving the hostname mismatch failure issue that arises without it by passing the 'correct' host to the TLS handshake.

import boto3
from botocore.auth import SigV4Auth, _host_from_url
from botocore.awsrequest import AWSRequest
from botocore.credentials import Credentials
from gremlin_python.driver.aiohttp.transport import AiohttpTransport
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
from gremlin_python.process.anonymous_traversal import traversal

# # If Jupyter notebook
# import nest_asyncio
# nest_asyncio.apply()


def get_signed_headers(url: str, region: str, creds: Credentials) -> dict:
    request = AWSRequest(method="GET", url=url)
    request.headers.add_header("Host", _host_from_url(str(request.url)))
    SigV4Auth(creds, "neptune-db", region).add_auth(request)
    return dict(request.headers)


neptune_hostname = "mycluster.eu-west-2.neptune.amazonaws.com"
true_database_url = f"wss://{neptune_hostname}:8182/gremlin"
local_database_url = "wss://localhost:8182/gremlin"

creds = boto3.Session(profile_name="my_profile").get_credentials()
headers = get_signed_headers(true_database_url, "eu-west-2", creds)

conn = None

try:
    conn = DriverRemoteConnection(
        local_database_url,
        "g",
        headers=headers,
        transport_factory=lambda: AiohttpTransport(server_hostname=neptune_hostname),
    )

    g = traversal().withRemote(conn)

    # Return 1 if connection successful
    print(g.inject(1).toList())
finally:
    if conn:
        conn.close()

import boto3
from botocore.auth import SigV4Auth, _host_from_url
from botocore.awsrequest import AWSRequest
from botocore.credentials import Credentials
from gremlin_python.driver.aiohttp.transport import AiohttpTransport
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
from gremlin_python.process.anonymous_traversal import traversal

# # If Jupyter notebook
# import nest_asyncio
# nest_asyncio.apply()


def get_signed_headers(url: str, region: str, creds: Credentials) -> dict:
    request = AWSRequest(method="GET", url=url)
    request.headers.add_header("Host", _host_from_url(str(request.url)))
    SigV4Auth(creds, "neptune-db", region).add_auth(request)
    return dict(request.headers)


neptune_hostname = "mycluster.eu-west-2.neptune.amazonaws.com"
true_database_url = f"wss://{neptune_hostname}:8182/gremlin"
local_database_url = "wss://localhost:8182/gremlin"

creds = boto3.Session(profile_name="my_profile").get_credentials()
headers = get_signed_headers(true_database_url, "eu-west-2", creds)

conn = None

try:
    conn = DriverRemoteConnection(
        local_database_url,
        "g",
        headers=headers,
        transport_factory=lambda: AiohttpTransport(server_hostname=neptune_hostname),
    )

    g = traversal().withRemote(conn)

    # Return 1 if connection successful
    print(g.inject(1).toList())
finally:
    if conn:
        conn.close()

Jump to solution

Neptune Local Development Access Methods

Neptune Local Development Access Methods

Similar Threads

Similar Threads

Similar Threads