Lil Villa Comments - Answer Overflow

plugins {
    id 'myproject.scala-conventions'
}

dependencies {
    implementation "org.apache.hadoop:hadoop-aws:${hadoopVersion}"
    implementation "org.apache.hadoop:hadoop-common:${hadoopVersion}"
    implementation "org.apache.hadoop:hadoop-client:${hadoopVersion}"
}

plugins {
    id 'myproject.scala-conventions'
}

dependencies {
    implementation "org.apache.hadoop:hadoop-aws:${hadoopVersion}"
    implementation "org.apache.hadoop:hadoop-common:${hadoopVersion}"
    implementation "org.apache.hadoop:hadoop-client:${hadoopVersion}"
}

9 replies

JCHJava Community | Help. Code. Learn.

•Created by Lil Villa on 5/29/2023 in #java-help

Can't resolve transitive dependencies by myself

main flink gradle file:

import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar

plugins {
    id 'application'
    id 'myproject.scala-conventions'
    id 'com.github.johnrengelman.shadow'
}

sourceSets.main.compileClasspath += configurations.flinkShadowJar
sourceSets.main.runtimeClasspath += configurations.flinkShadowJar
sourceSets.test.compileClasspath += configurations.flinkShadowJar
sourceSets.test.runtimeClasspath += configurations.flinkShadowJar
run.classpath = sourceSets.main.runtimeClasspath

dependencies {
        implementation "org.apache.flink:flink-streaming-scala_${scalaVersion}:${flinkVersion}"
        implementation "org.apache.flink:flink-clients:${flinkVersion}"
        flinkShadowJar project(':commons')
        flinkShadowJar 'org.apache.flink:flink-connector-kinesis:1.15.2'
        flinkShadowJar "org.apache.hadoop:hadoop-aws:${hadoopVersion}"
}


tasks.withType(ShadowJar).configureEach {
    zip64 true
    from project.sourceSets.main.runtimeClasspath
    configurations = [ project.configurations.flinkShadowJar ]
    // https://github.com/johnrengelman/shadow/blob/master/src/main/groovy/com/github/jengelman/gradle/plugins/shadow/ShadowJavaPlugin.groovy#L86
    exclude('META-INF/INDEX.LIST', 'META-INF/*.SF', 'META-INF/*.DSA', 'META-INF/*.RSA', 'module-info.class')
}

import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar

plugins {
    id 'application'
    id 'myproject.scala-conventions'
    id 'com.github.johnrengelman.shadow'
}

sourceSets.main.compileClasspath += configurations.flinkShadowJar
sourceSets.main.runtimeClasspath += configurations.flinkShadowJar
sourceSets.test.compileClasspath += configurations.flinkShadowJar
sourceSets.test.runtimeClasspath += configurations.flinkShadowJar
run.classpath = sourceSets.main.runtimeClasspath

dependencies {
        implementation "org.apache.flink:flink-streaming-scala_${scalaVersion}:${flinkVersion}"
        implementation "org.apache.flink:flink-clients:${flinkVersion}"
        flinkShadowJar project(':commons')
        flinkShadowJar 'org.apache.flink:flink-connector-kinesis:1.15.2'
        flinkShadowJar "org.apache.hadoop:hadoop-aws:${hadoopVersion}"
}


tasks.withType(ShadowJar).configureEach {
    zip64 true
    from project.sourceSets.main.runtimeClasspath
    configurations = [ project.configurations.flinkShadowJar ]
    // https://github.com/johnrengelman/shadow/blob/master/src/main/groovy/com/github/jengelman/gradle/plugins/shadow/ShadowJavaPlugin.groovy#L86
    exclude('META-INF/INDEX.LIST', 'META-INF/*.SF', 'META-INF/*.DSA', 'META-INF/*.RSA', 'module-info.class')
}

9 replies

JCHJava Community | Help. Code. Learn.

•Created by Lil Villa on 5/29/2023 in #java-help

Can't resolve transitive dependencies by myself

Dependencies output with every other dependency which does not have guava as a dependency excluded -> https://gist.github.com/tomazvila/1837a33228cecdb9fbc2e7985eee94e3

9 replies

JCHJava Community | Help. Code. Learn.

•Created by Lil Villa on 5/4/2023 in #java-help

Flink ShutdownHookManager throws Trying to access closed classloader when destroying itself

Resolved my problem by adding:

Thread.currentThread().setContextClassLoader(ClassLoader.getSystemClassLoader)

Thread.currentThread().setContextClassLoader(ClassLoader.getSystemClassLoader)

To my sink. I think it forces my job to use System class loader instead of dynamic flink class loader which somewhere and somewhy was being removed when shutdown executor tries to use it for its own shutdown.

8 replies

JCHJava Community | Help. Code. Learn.

•Created by Lil Villa on 11/29/2022 in #java-help

Force Spring Yaml configuration environment parameter to be treated as yaml instead of String

Thank you for an answer :prayge:

7 replies

JCHJava Community | Help. Code. Learn.

•Created by Lil Villa on 10/11/2022 in #java-help

how to add Log4J json template layout

This is just what I needed! Thank you @imp_o_rt :prayge:

6 replies

Gaming

Programming