@@ -34,17 +34,18 @@ fun main() {
3434}
3535```
3636
37- ## Building the application with Maven
38-
37+ ## Building the application
3938Because Kotlin Spark API is not part of the official Apache Spark distribution yet, it is not enough to add Spark
40- as a dependency in your pom.xml file.
39+ as a dependency to your build file.
4140You need to:
4241- Add Spark as a dependency
4342- Add Kotlin Spark API as a dependency
4443- Add Kotlin Standard Library as a dependency
4544
4645When packaging your project into a jar file, you need to explicitly include Kotlin Spark API and Kotlin Standard Library
47- dependencies, for example, using ` maven-shade-plugin ` .
46+ dependencies. Here you can find an example of building your application with Maven, and with Gradle.
47+
48+ ### Building the application with Maven
4849
4950Here's what the ` pom.xml ` looks like for this example:
5051``` xml
@@ -148,17 +149,55 @@ Here's what the project structure should look like:
148149
149150Now you can package the application using Maven:
150151` mvn package `
151-
152- When done, you can execute the packaged application with ` ./bin/spark-submit ` :
153152
154- ` YOUR_SPARK_HOME/bin/spark-submit --class "SimpleApp" --master local YOUR_PROJECT/target/kotlin-spark-example-1.0-SNAPSHOT.jar `
153+ ### Building the application with Gradle
155154
156- This example is also available as a [ GitHub repo ] ( https://github.com/MKhalusova/kotlin-spark-example ) , feel free to give it a try.
155+ Here's what the ` build.gradle ` looks like for this example:
157156
157+ ```
158+ plugins {
159+ id 'org.jetbrains.kotlin.jvm' version '1.3.72'
160+ id 'com.github.johnrengelman.shadow' version '5.2.0'
161+ }
158162
163+ group = 'org.example'
164+ version = '1.0-SNAPSHOT'
159165
160-
166+ repositories {
167+ mavenCentral()
168+ maven { url = 'https://jitpack.io' }
169+ }
161170
171+ dependencies {
172+ // Kotlin stdlib
173+ implementation 'org.jetbrains.kotlin:kotlin-stdlib:1.3.72'
174+ // Kotlin Spark API
175+ implementation 'com.github.JetBrains.kotlin-spark-api:kotlin-spark-api:0.1.0'
176+ // Apache Spark
177+ compileOnly 'org.apache.spark:spark-sql_2.12:3.0.0-preview2'
178+ }
179+
180+ compileKotlin {
181+ kotlinOptions.jvmTarget = '1.8'
182+ }
183+
184+ shadowJar {
185+ dependencies {
186+ exclude(dependency {
187+ it.moduleGroup == 'org.apache.spark' || it.moduleGroup == "org.scala-lang"
188+ })
189+ }
190+ }
191+ ```
192+
193+ Now you can package the application using Gradle:
194+ ` gradle shadowJar `
162195
163196
164-
197+ ## Executing the application with spark-submit
198+
199+ Once you have your jar, you can execute the packaged application with ` ./bin/spark-submit ` :
200+
201+ ` YOUR_SPARK_HOME/bin/spark-submit --class "SimpleApp" --master local [path to your jar] `
202+
203+ This example is also available as a [ GitHub repo] ( https://github.com/MKhalusova/kotlin-spark-example ) , feel free to give it a try.
0 commit comments