Analyzing samples of Gremlin Queries in Neptune Notebook
Hey everyone,
I’m working on a project where we give internal customers access to our Neptune graph through Neptune Notebook. There are already quite a few users, and we want to analyze the queries they run to see which parts of our ontology are used more and which parts are less utilized. This is not as straight-forward as retrieving all labels from the query, since our edge labels are not unique, and if people would be using
We’ve figured out how to override the Gremlin magic in Neptune Notebook to add our custom logic to handle each query. And for my problem I’m considering two approaches:
- Running the Gremlin profiler on each query to get detailed info on the nodes visited and then applying custom language analysis algorithms.
- Collecting this data and feeding it into an LLM to summarize the queries and answer questions I'm interested in.
Has anyone here done something similar or have any insights on this? Would love to hear about your experiences or any advice you might have!
I’m working on a project where we give internal customers access to our Neptune graph through Neptune Notebook. There are already quite a few users, and we want to analyze the queries they run to see which parts of our ontology are used more and which parts are less utilized. This is not as straight-forward as retrieving all labels from the query, since our edge labels are not unique, and if people would be using
.in.outg.V()g.V()We’ve figured out how to override the Gremlin magic in Neptune Notebook to add our custom logic to handle each query. And for my problem I’m considering two approaches:
- Running the Gremlin profiler on each query to get detailed info on the nodes visited and then applying custom language analysis algorithms.
- Collecting this data and feeding it into an LLM to summarize the queries and answer questions I'm interested in.
Has anyone here done something similar or have any insights on this? Would love to hear about your experiences or any advice you might have!
Solution
I think this is going to depend on how granular you want to get. If the intent is to see what labeled vertices or edges are accessed, then just looking at a query in the audit log would be sufficient. But, if your intent is to see every atomic component that is accessed in the database as part of query execution, that could be expensive. It is possible, though. You could run every query through the Neptune Gremlin Profiler: https://docs.aws.amazon.com/neptune/latest/userguide/gremlin-profile-api.html and set
With the list of indexed lookup patterns, you could possibly maintain an external counter (maybe in sorted set in Redis/Valkey) with a a key of the S-P-O-G combination and the value being the number of times accessed.
Just be aware that attaining a Neptune Gremlin Profile output requires that you run the query again. So you may not be able to use this to capture writes (without rewriting the data) and it will incur additional database resources to re-run all of the read queries.
profile.indexOpsTrueWith the list of indexed lookup patterns, you could possibly maintain an external counter (maybe in sorted set in Redis/Valkey) with a a key of the S-P-O-G combination and the value being the number of times accessed.
Just be aware that attaining a Neptune Gremlin Profile output requires that you run the query again. So you may not be able to use this to capture writes (without rewriting the data) and it will incur additional database resources to re-run all of the read queries.
The Gremlin profile feature in Neptune runs a specified traversal, collects metrics about the run, and produces a profile report.
