Plans for the Hadoop integration

Notice https://docs.janusgraph.org/advanced-topics/hadoop/ from JanusGraph documentation about JanusGraph and Hadoop integration, would like to know where we could get more information on the project plan/status. Thanks,
4 Replies
Florian Hockmann
Florian Hockmann•2y ago
This was originally asked by @chiujl in our previous questions chat: https://discord.com/channels/981533699378135051/982256175196295180/1110320366368346203 Hi @chiujl, the story with JanusGraph's Hadoop integration is that JanusGraph mostly relies on TinkerPop's Hadoop-Gremlin to execute Gremlin traversals or so called Vertex Programs on Spark. JanusGraph mostly adds the functionality to read data from your storage backend, like Cassandra or HBase, to make it available for Hadoop/Spark. So, what you can do with that is mostly described in the TinkerPop reference docs: Hadoop-Gremlin: https://tinkerpop.apache.org/docs/current/reference/#hadoop-gremlin GraphComputer: https://tinkerpop.apache.org/docs/current/reference/#graphcomputer
paritoshgote
paritoshgote•2y ago
@HadoopMarc , @florianhockmann , @boxuanli : We are running into an issue while launching the gremlin.sh with hadoop (ref: https://tinkerpop.apache.org/docs/current/reference/#hadoop-gremlin) using janus. Would you folks have any tips for this issue? Error with Janus 0.6.3: 20:00:36 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: wasbs I tried using the latest janus 1.0.0-rc2 as well, and that gave a different error Error with Janus 1.0.0-rc2: 21:32:16 WARN org.apache.hadoop.util.NativeCodeLoader.<clinit> - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found
sshuser@hn0-spark:~/janusgraph/janusgraph-1.0.0-rc2$ bin/gremlin.sh

plugin activated: tinkerpop.tinkergraph
21:32:16 INFO org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.getCustomizers - HADOOP_GREMLIN_LIBS is set to: /home/sshuser/janusgraph/janusgraph-1.0.0-rc2/lib
21:32:16 WARN org.apache.hadoop.util.NativeCodeLoader.<clinit> - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found
at org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.lambda$static$0(HadoopGremlinPlugin.java:125)

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2665)
... 67 more
sshuser@hn0-spark:~/janusgraph/janusgraph-1.0.0-rc2$ bin/gremlin.sh

plugin activated: tinkerpop.tinkergraph
21:32:16 INFO org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.getCustomizers - HADOOP_GREMLIN_LIBS is set to: /home/sshuser/janusgraph/janusgraph-1.0.0-rc2/lib
21:32:16 WARN org.apache.hadoop.util.NativeCodeLoader.<clinit> - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found
at org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.lambda$static$0(HadoopGremlinPlugin.java:125)

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2665)
... 67 more
Florian Hockmann
Florian Hockmann•2y ago
@paritoshgote Could you please create a dedicated post for your question? Having only one question per post makes it much easier to follow
paritoshgote
paritoshgote•2y ago
@florianhockmann : Created a separate post here https://discord.com/channels/981533699378135051/1113947036664995880, thanks 🙂
Discord
Discord - A New Way to Chat with Friends & Communities
Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.
Want results from more Discord servers?
Add your server