Can `CqlInputFormat` do predicate pushdowns/query based prefilters?
Hi! First of all, thank you all for your work on JanusGraph.
In my use case, I have a medium-large graph, ~3TB currently, might be 1-2 orders of magnitude bigger later. The data in it is generally clustered in a time-based fashion, e.g. newer vertices are mostly connected to other newer vertices (a timestamp is stored as a vertex property).
I am writing an OLAP pipeline with Spark where JanusGraph, backed by Cassandra, is the source, and I use Tinkerpop's
A very common operation that I'm going to have to do, based on the above clustering assumption, is pre-filtering vertices by a timestamp range before running my logic on the subgraph. As an example, I would like to, say, download the last couple days' worth of vertices on my laptop for running some tests. Per my understanding, currently this would entail unconditionally loading the entire dataset in the Spark cluster's memory every time. Is that correct? Is there an alternative?
I have looked into
Thank you!
cc @criminosis
In my use case, I have a medium-large graph, ~3TB currently, might be 1-2 orders of magnitude bigger later. The data in it is generally clustered in a time-based fashion, e.g. newer vertices are mostly connected to other newer vertices (a timestamp is stored as a vertex property).
I am writing an OLAP pipeline with Spark where JanusGraph, backed by Cassandra, is the source, and I use Tinkerpop's
hadoop-gremlinCqlInputFormatA very common operation that I'm going to have to do, based on the above clustering assumption, is pre-filtering vertices by a timestamp range before running my logic on the subgraph. As an example, I would like to, say, download the last couple days' worth of vertices on my laptop for running some tests. Per my understanding, currently this would entail unconditionally loading the entire dataset in the Spark cluster's memory every time. Is that correct? Is there an alternative?
I have looked into
CqlInputFormatWHEREThank you!
cc @criminosis
