Setup Spark with IntelliJ IDEA on Linux Mint 20

Installing IntelliJ IDEA

To start we need an IDE that supports Spark. I went with IntelliJ. Just head over to the IntelliJ Website and choose the community edition.

Setting up the environment

Now that that’s out of the way we need to add the Scala extension.

libraryDependencies ++=
Seq("org.apache.spark" %% "spark-core" % "3.1.1",
"org.apache.spark" %% "spark-sql" % "3.1.1")
import org.apache.spark.sql._
import org.apache.log4j._
Logger.getLogger("org").setLevel(Level.ERROR)

Additional Tips:

You can create a directory under the main project and call it data in order to access data files with a simple “data/YourDataFile”

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store