Is it possible to configure SSL with PEM certificate types?
Hi all, I'm new to this group and currently working getting an implementation of Gremlin (Aerospike Graph) to listen over SSL.
The certificates we get from our provider's API are only served in PEM format. It appears, according to the documentation that the
keyStoreType and trustStoreType either JKS or PKCS12 format: https://tinkerpop.apache.org/javadocs/current/full/org/apache/tinkerpop/gremlin/server/Settings.SslSettings.html
Is this true? Is there any way for us to configure SSL with PEM format certificates?...Solution:
Hi @joshb, am I correct in assuming you are using the Java driver to connect to Aerospike? The java driver uses the JSSE keyStore and trustStore, which as far as I understand does not support the PEM format. You may be able to use a 3rd party tool such as openssl to convert from PEM to PKCS12 (https://docs.openssl.org/1.1.1/man1/pkcs12/).
Perhaps @aerospike folks may have more direct recommendations for driver configuration....
Query works when executed in console but not in javascript
```
const combinedQuery = gremlin.V(profileId)
.project('following', 'follows')
.by(
__.inE('FOLLOWS').outV().dedup().id().fold()...
Very slow regex query (AWS Neptune)
We have a query that searches a data set of about ~400,000 vertices, matching properties using a case insensitive
TextP.regex() expression. We are observing very bad query performance; even after several other optimizations, it still takes 20-45 seconds, often timing out.
Simplified query:
```
g.V()...Solution:
When Neptune stores data it stores it in 3 different indexed formats (https://docs.aws.amazon.com/neptune/latest/userguide/feature-overview-data-model.html#feature-overview-storage-indexing), each of which are optimized for a specific set of common graph patterns. Each of these indexes is optimized for exact match lookups so when running queries that require partial text matches, such as a regex query, all the matching property data needs to be scanned to see if it matches the provided expression.
To get a performant query for partial text matches the suggestion is to use the Full Text search integration (https://docs.aws.amazon.com/neptune/latest/userguide/full-text-search.html) , which will integrate with OpenSearch to provide robust full text searching capabilities within a Gremlin query...
To get a performant query for partial text matches the suggestion is to use the Full Text search integration (https://docs.aws.amazon.com/neptune/latest/userguide/full-text-search.html) , which will integrate with OpenSearch to provide robust full text searching capabilities within a Gremlin query...
How can we extract values only
"Latitude",
{
"@type": "g:Double",
"@value": 45.2613104
},...
Solution:
If I understand this correctly, you are first trying to take the result of a gremlin query that returns Latitude and Longitude (like in the initial post you made), and use those values in the in the math() step that calculates the Haversine formula (your latest post). If that is the case, you have two options.
1. You should combine this into one Gremlin query. You can save the results of the Latitude and Longitude to variables or use them in a by() modulator to the math step. Assuming that those values are properties on a vertex called 'lat' and 'lon' it would look something like
g.V().project('Latitude', 'Longitude').by('lat').by('lon').math(...). You would replace the ... in the math() step with the Haversine formula.
2. If you want to keep these as two separate queries, then you should use one of the Gremlin Languages Variants (GLVs) which are essentially drivers that will automatically deserialize the result into the appropriate type so you don't have to deal with the GraphSON (which is what your initial post shows). Read triggan's answer above for more details about that....How to speed up gremlin query
Hi, I am working with Janusgraph and my query is taking a while to execute (around 2.8 seconds), but I would like it to be faster. I read that I should create a composite index to improve speed and performance or something of that sort, but I am unfamiliar with how to do that in Python.
Here is my query: g.V().has("person", "name", "Bob").outE("knows").has("weight", P.gte(0.5)).inV().values("name").toList()
What my query does is it finds all the nodes that Bob has relationship "knows" with, as long as the weight of the edge to those nodes are >=0.5. Bob has around ~600 nodes that it's connected to with the "knows" relationship. It's fairly slow and takes 2.5-2.8 secs to complete....
Solution:
I read that I should create a composite index to improve speed and performance or something of that sort, but I am unfamiliar with how to do that in Python.as a point of clarification around indices, you wouldn't likely do that step with python. you typically use gremlinpython just to write Gremlin. For index management, you need to use JanusGraph's APIs directly. often those are just commands you would execute directly in Gremlin Console against a JanusGraph instance....
logging and alerting inside a gremlin step
I am trying to add a log statement inside the step but I am getting an error Server error: null (599)
ResponseError: Server error: null (599)
This is the error we get // Log the creation of a shell flight
.sideEffect(() => logger.info(
Shell flight created for paxKey ${paxKey} with flightId ${flightLegID}))
...Solution:
Since you are getting a "Server error" I assume you are using Gremlin in a remote context. I'd further assume you are not sending a script to the server, but are using bytecode or are using a graph like Amazon Neptune. Those assumptions all point to the fact that you can't use a lambda that way in remote contexts.
The approach you are using to write your lambda is for embedded use cases only (i.e. where query is executed in the same JVM where it was created). If you want to send a lambda remotely you would need to have a server that supports them (e.g. Neptune does not, but Gremlin Server with Groovy ScriptEngine processing does) and then follow these instructions: https://tinkerpop.apache.org/docs/current/reference/#gremlin-java-lambda
The other thing to consider is that the lambda will be executed remotely so it might not know what "logger" is and the log message will appear on the server note the client. ...
Optimizing connection between Python API (FastAPI) and Neptune
Hi guys. I've been working with gremlin python in my company for the past 4 years, using Neptune as the database. We are running a FastAPI server, where Neptune has been the main database since the beginning.
We always have been struggling to get a good performance on the API, but recently it has become a more latent pain, with endpoints taking more than 10s to respond.
We took some actions trying to improve this perfomance, such as updating the cluster to the latest engine version, and the same for FastAPI and gremlin-python dependencies.
...
Solution:
There's a lot to unpack here....
1. We state in our docs that t4g.medium instances are really not great for production workloads. We support them for initial development so users can keep cost down, but the amount of resources available, and the fact that they are burstable instances, really constrains their usability. Once you've used up CPU credits, you're going to get throttled.
2. Neptune's concurrency model is based on instance size and the number of vCPUs per instance. For each vCPU, there are two query execution threads. So on a t4g.medium or an r6g.large instance, there are 2 vCPUs. That means that instance can only be computing 4 concurrent requests at a time. If you need more concurrency, then you should look to scale to a larger instance with more vCPUs. If you're workload varies over time, you may want to investigate using Neptune Serverless, which can automatically scale vertically to meet the needs of the application. There's a good presentation from last year's re:Invent that discusses when Serverless works best and when not to use it: https://youtu.be/xAdWa0Ahiok?si=OeSe-_L3ErcYH-XU...
## Breadth-First Traversal Fact Check
Using the Neptune Query Profiling, I have found out that Gremlin queries seems to use depth first strategy to search things and as a result it tends to be both time and resource intensive especially when what I am looking for is a node just a 1 or 2 levels below.
To do a Breadth-First Traversal the following approach has been suggested, but not sure if this really does the trick.
If my goal is to find nearest nodes quickly, what could be efficient approaches?...
Solution:
Neptune uses BFS as the default traversal strategy. You can change the method in which a
repeat() is executed via the query hint as noted here: https://docs.aws.amazon.com/neptune/latest/userguide/gremlin-query-hints-repeatMode.htmlHow to find the edges of a node that have a weight of x or greater?
I have a graph with nodes that are connected to one another, and I have weights on the edges that connect these nodes.
On this query g.V().has("person", "name", "A").out("is friends with").values("name").to_list() returns the names of people that person "A" has the relation "is friends with". But I would like to filter by the weight value. Person A has a weight of 0.9 with Person B, and a weight of 0.7 with Person C. I would like to only get back the people person A has an edge with a weight of greater than 0.8, how can I do that? I am using gremlin-python. I have tried g.V().has("person", "name", "A").out("is friends with").has("weight", gte(0.8)) but I get an error saying NameError: name "gte" is not defined. Did you mean: 'g'?...
Solution:
g.V().has("person", "name", "A").out(... returns Vertices
to get edges need to use outE()
something like
a_friends = g.V().has("person", "name", "A").outE("is friends with").has("weight", P.gt(0.75)).inV().to_list()...op_traversal P98 Spikes
Hi TinkerPop team!
I'm observing these abrupt spikes in my gremlin server P98 metrics in my JanusGraph environment. I've been looking at the TraversalOpProcessor (https://github.com/apache/tinkerpop/blob/master/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/op/traversal/TraversalOpProcessor.java) code over the last couple days for some ideas of what could be causing it but I'm not seeing an obvious smoking gun so figured I'd ask around.
The traversal in question is being submitted from Rust via Bytecode using gremlin-rs. Specifically with some additions I've made that added mergeV & mergeE among other things. It's in a PR awaiting the maintainer's review, but that's not really relevant to my question, but over here if you want to see it: https://github.com/wolf4ood/gremlin-rs/pull/214. The GraphSONV3 serializer is what's being used by the library....
Solution:
For anyone else that finds this thread the things I ended up finding to be issues:
- Cassandra's disk throughput I/O (EBS gp3 is 125MB/s by default, at least for my use case a I was periodically maxxing that out, increasing to 250MB/s resolved that apparent bottleneck). So if long sustained writing occured the 125MB/s was not sufficent.
- Optimizing traversals to using mergeE/mergeV that were either older groovy-script based evaluations I was submitting or older
fold().coalesce(unfold(),...) style "get or create" based vertex mutations....
neo4j-gremlin not working with JDK17
Neo4jGraph fails to start with top-level error like:
Error starting org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactorywith causes like:
Suppressed: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.scheduler.CentralJobScheduler@5906ebfb' failed to transition from stopped to shutting_down. Please see the attached cause exception "Exception java.lang.LinkageError: Could not get Throwable message field [in thread "main"]"and:...
Solution:
This issue relates back to: https://tinkerpop.apache.org/docs/current/upgrade/#_building_and_running_with_jdk_17 and the need to add this JVM option:
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED...
I am unsure on how to use Python to add graphs to JanusGraph
I am having a difficult time using python-gremlin. I am really unsure as to how I can create a graph, create vertices and edges, and then upload it to the database. Could someone provide a guide on how to do these things? I followed the JanusGraph tutorial as well as the Tinkerpop tutorials on how to use gremlin-python but nothing seems to be working for me.
Solution:
the key bit to understand with gremlin-python is that you can only use it to query/mutate a graph with the Gremlin language and that graph must be hosted in Gremlin Server (or be compliant with its protocol, like Amazon Neptune). other functions like, "create a graph" or access provider specific functions like creating indices are not possible with the Gremln language and therefore not possible iwth gremlin-python (or other non-JVM programming languages that support Gremln).
if you are wholly new to TinkerPop and JanusGraph (and maybe graph databases themselves), my recommendation would be to not start with JanusGraph. It's natural to want to dive right in to the graph you want to use, but I think it's better to take a slower approach.
i think the learning process is like:...
Optimising python-gremlin for fastApi
PS: Please give me some rope here as I am new to gremlin-python!
I have a fastAPI which is serving some Rest API calls based on the data that I am fetching from a Neptune instance. I am using gremlin-python library to make the queries to graph instance.
Some points to note:...
C# Profiling Duration
Does anyone have an example of how to get the millisecond/microsecond execution time of a query in C#? I can't find any examples of this online.
Solution:
I don't have really good alternative.
Option is to run same traversal from Gremlin Console or java GLV.
If you have GremlinGroovyScriptEngine (it's default for TinkerPop 3.6-3.7) then workaround is to convert response to string or Map before sending back from server, something like
await client.SubmitWithSingleResultAsync<Object>("g.V().profile().toList().toString()");...Expecting java.lang.ArrayList/java.lang.List instead of java.lang.String. Where am I going wrong?
```
gremlin> :remote connect tinkerpop.server conf/remote.yaml session
Jun 19, 2024 6:04:36 PM org.yaml.snakeyaml.internal.Logger warn
WARNING: Failed to find field for org.apache.tinkerpop.gremlin.driver.Settings.serializers
18:04:37 INFO org.apache.tinkerpop.gremlin.driver.Connection.<init> - Created new connection for ws://localhost:8182/gremlin...
Solution:
A few things you should know based on your needs:
1. with JanusGraph you can't store
Map or List so that may be a blocker for you
2. multi-properties aren't List - they are multiple values stored under the same property key, so those semantics may be a blocker for you
3. Gremlin has no steps that can help you detect a type. Whatever type detection you would do, it would have to happen in the client application (technically what you are doing when you call next() and class) which may be a blocker for you.
4. Using just Gremlin It's hard to detect if you are using multiproperties or not because it has no knowledge of the schema. Maybe you could consult JanusGraph for the schema for the types. Counting properties of a key doesn't really work either because you could have a key with just one property and it still could be a multiproperty (with just one value)....Is NoHostAvailableException losing/not including relevant error context (3.7.0 above)?
Hey folks,
I've recently noticed that G.V() is "not as good" as it used to be in reporting some specific connectivity issue, and upon further investigation managed to attribute this to a change in behaviour with NoHostAvailableException seemingly losing some relevant context.
In my case, I'm testing the scenario of trying to connect to Azure Cosmos DB from an IP that is not allowed through their firewall, which usually results in a bespoke error message for Azure going as follows:
...
Invalid handshake response getStatus: 403 Request originated from IP 213.31.211.185 through public internet. This is blocked by your Cosmos DB account firewall settings. More info: https://aka.ms/cosmosdb-tsg-forbidden
Invalid handshake response getStatus: 403 Request originated from IP 213.31.211.185 through public internet. This is blocked by your Cosmos DB account firewall settings. More info: https://aka.ms/cosmosdb-tsg-forbidden
Solution:
i dont think we've changed any behavior for
NoHostAvailableException since 3.5.5: https://tinkerpop.apache.org/docs/current/upgrade/#_gremlin_driver_host_availability Since that time there is really only one way that an NHA is thrown: if the connection pool cannot initialize a connection to any host. we are selective in what exceptions are raised within the NHA because there are cases where the exception can be more confusing than helpful. in this case, we weren't including handshake exception...Setting index in gremlin-python
I was trying to create vote graph from tutorial on loading data in gremlin-python and afaik you can't simply add index from non-JVM languages because for example there is no TinkerGraph that you could
.open(). I don't know how better is performance when having index on 'userId' but my code simply takes too long go through queries from vote file.
I tried using client functionality
```py
ws_url = 'ws://localhost:8182/gremlin'...Solution:
if you simply edit that line of code to create your index and load your vote data, every time you start Gremlin Server it will have that all setup and ready to go.
Anyone had issues with gremlinpython driver async_timeout?
This seems to be an undocumented and unrequired (but actually required) library, every time i go to use the python driver in a cloud computer that is fresh, I have to install gremlinpython then async_timeout. This makes sense because it's not a default python library, but seems to be for some reason on the github machiens tinkerpop uses to test.
I am wondering if anyone else has noticed this and if we should perhaps put it in the https://github.com/apache/tinkerpop/blob/master/gremlin-python/src/main/python/setup.py or something....
Solution:
Well looks like its just me so i wont dig into fixing this.
Dynamically calculated values from last vertex to select outgoing edges for further traversal
Hi - I asked the question on SO (https://stackoverflow.com/questions/78611365/neptune-graph-traversal-that-uses-dynamically-calculated-values-from-last-vertex) but there is no answer. I am really interested if this is a scenario gremlin can handle. I am not very well versed with Tinkerpop so I am not sure whether I am just trying to build a query that is complex, or asking for a case that is just not supported in tinkerpop. I work for a company that is seriously considering onboarding onto greml...
Window functions in gremlin
Is there any way to apply window functions in gremlin queries I need to convert the following SQL query into gremlin.
SELECT ass.id, fin.id, ...