A place you can learn spark from basics
How to Install in Mac: Basic set up requires
- Install Apache Spark using Homebrew. a. Install Homebrew if you don’t have it already by entering this from a terminal prompt: /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" b. Enter brew install apache-spark c. Create a log4j.properties file via cd /usr/local/Cellar/apache-spark/2.0.0/libexec/conf cp log4j.properties.template log4j.properties (substituted 2.0.0 for the version actually installed) d. Edit the log4j.properties file and change the log level from INFO to ERROR on log4j.rootCategory.
- Install the Scala IDE from http://scala-ide.org/download/sdk.html
- Test it out! a. Cd to the directory apache-spark was installed to (such as /usr/local/Cellar/apachespark/2.0.0/libexec/) and then ls to get a directory listing. b. Look for a text file we can play with, like README.md or CHANGES.txt c. Enter spark-shell d. At this point you should have a scala> prompt. If not, double check the steps above. e. Enter val rdd = sc.textFile(“README.md”) (or whatever text file you’ve found) f. Enter rdd.count() g. You should get a count of the number of lines in that file! Congratulations, you just ran your first Spark program! h. Hit control-D to exit the spark shell, and close the console window i. You’ve got everything set up! Hooray!
If you donot set up your Environment variables cd /usr/local/Cellar/apache-spark/2.1.0/libexec spark-shell The above syntax will get you into scala shell.
** - COPY ALL THE LIB FILES FROM JARS FOLDER INSIDE libexec folder into your project folder to get visibility for Scala Eclipse to add external jars under properties inorder to run a scala program