Plans for the Hadoop integration
Notice https://docs.janusgraph.org/advanced-topics/hadoop/ from JanusGraph documentation about JanusGraph and Hadoop integration, would like to know where we could get more information on the project plan/status. Thanks,
4 Replies
This was originally asked by @chiujl in our previous questions chat: https://discord.com/channels/981533699378135051/982256175196295180/1110320366368346203
Hi @chiujl, the story with JanusGraph's Hadoop integration is that JanusGraph mostly relies on TinkerPop's Hadoop-Gremlin to execute Gremlin traversals or so called Vertex Programs on Spark. JanusGraph mostly adds the functionality to read data from your storage backend, like Cassandra or HBase, to make it available for Hadoop/Spark.
So, what you can do with that is mostly described in the TinkerPop reference docs:
Hadoop-Gremlin: https://tinkerpop.apache.org/docs/current/reference/#hadoop-gremlin
GraphComputer: https://tinkerpop.apache.org/docs/current/reference/#graphcomputer
@HadoopMarc , @florianhockmann , @boxuanli :
We are running into an issue while launching the gremlin.sh with hadoop (ref: https://tinkerpop.apache.org/docs/current/reference/#hadoop-gremlin) using janus. Would you folks have any tips for this issue?
Error with Janus 0.6.3:
20:00:36 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: wasbs
I tried using the latest janus 1.0.0-rc2 as well, and that gave a different error
Error with Janus 1.0.0-rc2:
21:32:16 WARN org.apache.hadoop.util.NativeCodeLoader.<clinit> - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found
@paritoshgote Could you please create a dedicated post for your question? Having only one question per post makes it much easier to follow
@florianhockmann : Created a separate post here https://discord.com/channels/981533699378135051/1113947036664995880, thanks 🙂
Discord
Discord - A New Way to Chat with Friends & Communities
Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.