- Install JDK: https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html (JDK 8, the first link on the page).
- Install sbt:
Mac: http://www.scala-sbt.org/release/docs/Installing-sbt-on-Mac.html (you can use
brew, it will install the latest). Win: https://www.scala-sbt.org/release/docs/Installing-sbt-on-Windows.html - Install Intellij: Mac: https://www.jetbrains.com/idea/download/#section=mac Win: https://www.jetbrains.com/idea/download/#section=windows
- Install Scala plugin for IJ: Intellij -> Preferences -> Plugins -> Search for official Scala plugin.
- Clone this repo:
git clone https://github.com/katrinsharp/spark-sample-project.git
- Open up a CLI (command line) and
cdto root folder of this repo. Type:
> sbt
You should see > after it's updating some its dependencies (Use exit for quitting). Now type:
> console
You should see scala> after some more updates (Use :q for quitting). You're ready to go with Scala CLI.
- Open the cloned project in IJ: File -> New -> Project from existing sources. You're ready to go with IJ.
sbt "testOnly sample.<name-of-the-test>" - executes specific test
sbt run - executing Main with Spark in-memory
sbt assembly - creates a self-contained jar: spark-sample-project/target/scala-2.11/spark-sample-project-assembly-0.1.jar that can be submitted to Spark cluster