A few lines of code to demo how streaming works with Spark, in particular using the extensions provided by Apache Bahir to read a live stream of tweets, which will be processed to assign it a sentiment score (using a very naive algorithm).

To make it work on your installation, be sure to add a twitter4j.properties under src/main/resources that includes the following information:

oauth.consumerKey=***
oauth.consumerSecret=***
oauth.accessToken=***
oauth.accessTokenSecret=***

Visit apps.twitter.com to get your own API keys.

To submit the job to an existing Spark installation you can package the job with the following command:

sbt package

and then submit it with the following command:

$SPARK_HOME/bin/spark-submit \
  --master $SPARK_MASTER \
  --jars $DEPENDENCIES \
  --class me.baghino.spark.streaming.twitter.example.TwitterSentimentScore \
  target/scala-2.11/spark-twitter-stream-example-assembly-1.0.0.jar

The Spark classpath should include org.apache.bahir:spark-streaming-twitter_2.11:2.0.1, org.twitter4j:twitter4j-core:4.0.4 and org.twitter4j:twitter4j-stream:4.0.4.

After running the sbt package command you'll find the required JARs in your local Ivy cache ($HOME/.ivy2/cache/).