How To Install Spark On Mac os

"Hello World, Hello Blog"

Posted by Leonard Yuan on April 30, 2018

How To Install Spark on Mac os

1. Introduction

Today, I will introduce the way to install spark on mac os. I will achieve the calculation of the data of MovieLens, it is the part of my undergraduate project. The topic of my undergraduate project is Design and Implementation of Recommendation System Based on Collaborative Filtering Algorithm. Because I choose MovieLens as my DataSet(190mb), so I have to use spark as analytics engine for large-scale data processing. Thus, it is neccessary to install Spark on mac os.

So, ler’s start.

2. Install Java JDK

We must install java JDK at first, we should download .dmg from official website, the link is shown as follow:

Download .dmg files

The website is shown like this:

Choose MAC OS X x64, then download it. When download process is finished, click .dmg files, and follow hints to finish install process.

Then you can use command java -version to confirm whether it is successful to install java JDK.

Aferwards, we start settle environment variables. First of all, we need to find profile files, its path is /etc/profile. We can use vim or something else to revise profile, we must add the path of java into profile, you should add following text:

export JAVA_HOME

After you save profile, use command source /etc/profile to make sure your revise to be valid. Finally, you can use following command to get the path of java:

$ echo $JAVA_HOME

From now, we can use java JDK on you mac os.

3. Install Scala

I will add details.

4. Install Spark

5. Introduction of Using Spark