JanusGraph on Azure HDInsights Spark Cluster
We are following the instructions here https://tinkerpop.apache.org/docs/current/reference/#hadoop-gremlin and are working on setting up the hadoop-gremlin on the spark cluster with JanusGraph 1.0.0-rc2. We are working through some of the issues with this integration where we needed to copy the two binaries in the Janus lib location
/usr/hdp/4.1.14.2/hadoop/hadoop-azure-3.1.1.4.1.14.2.jar
/usr/hdp/4.1.14.2/hadoop/azure-storage-7.0.1.jar
Now the current issue we are facing is the following. Would anyone know what might be going on, am I missing some more jar references here? Any tips/help greatly appreciated, thanks!
org.apache.hadoop.metrics2.impl.MetricsConfigException: Error creating plugin: org.apache.hadoop.metrics2.sink.WasbAzureIaasSink
at org.apache.hadoop.metrics2.impl.MetricsConfig.getPlugin(MetricsConfig.java:212) ~[hadoop-client-api-3.3.1.jar:?]
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.sink.WasbAzureIaasSink
at java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[?:1.8.0_362]
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProvider(Ljava/lang/String;Ljava/lang/String;)[C
at org.apache.hadoop.fs.azure.SimpleKeyProvider.getStorageAccountKey(SimpleKeyProvider.java:55)
at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.getAccountKeyFromConfiguration(AzureNativeFileSystemStore.java:1018)
@boxuanli , @florianhockmann , @HadoopMarc who have been helping on various channels regarding this effort.
/usr/hdp/4.1.14.2/hadoop/hadoop-azure-3.1.1.4.1.14.2.jar
/usr/hdp/4.1.14.2/hadoop/azure-storage-7.0.1.jar
Now the current issue we are facing is the following. Would anyone know what might be going on, am I missing some more jar references here? Any tips/help greatly appreciated, thanks!
org.apache.hadoop.metrics2.impl.MetricsConfigException: Error creating plugin: org.apache.hadoop.metrics2.sink.WasbAzureIaasSink
at org.apache.hadoop.metrics2.impl.MetricsConfig.getPlugin(MetricsConfig.java:212) ~[hadoop-client-api-3.3.1.jar:?]
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.sink.WasbAzureIaasSink
at java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[?:1.8.0_362]
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProvider(Ljava/lang/String;Ljava/lang/String;)[C
at org.apache.hadoop.fs.azure.SimpleKeyProvider.getStorageAccountKey(SimpleKeyProvider.java:55)
at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.getAccountKeyFromConfiguration(AzureNativeFileSystemStore.java:1018)
@boxuanli , @florianhockmann , @HadoopMarc who have been helping on various channels regarding this effort.
message.txt17.65KB
