This repo contains notebook exercises for a workshop teaching the best practices of using Spark for practicing data scientists in the context of a data scientist’s standard workflow. By leveraging Spark’s APIs for Python and R to present practical applications, the technology will be much more accessible by decreasing the barrier to entry.

There are corresponding videos on Youtube walking through and presenting the materials in here.

If you find any errors in the code or materials, please open a Github issue in this repository or send an email to [email protected]

Beginner to Intermediate

• Use Python for distributed computing
• Scale data processing with Spark
• Conduct exploratory data analysis with PySpark
• Utilize parallel computing with Ray
• Scale machine learning and artificial intelligence applications with Ray

This course is a good fit for anyone who needs to improve their fundamental understanding of scalable data processing with Python for use in machine learning or artificial intelligence applications.

• A basic understanding of programming in Python (variables, basic control flow, simple scripts).
• Familiarity with the vocabulary of data processing at scale, machine learning (dataset, training set, test set, model), and AI helpful but not required.

To run locally in a docker container 👇

make jupyter

or

docker run -p 8888:8888 -p 8265:8265 -p 8000:8000 -p 8089:8089 -v $(pwd):/home/jovyan/ --pull 'always' psychothan/scaling-data-science

Then open a web browser to the URL it spits out (the Jupyter server in the container uses token authentication)

Q: How can I find out all the methods that are available on DataFrame?

• In the IPython console type sales.[TAB]

• Autocomplete will show you all the methods that are available.

• To find more information about a specific method, say .cov type help(sales.cov)

• This will display the API documentation for that method.

Q: How can I find out more about Spark's Python API, MLlib, GraphX, Spark Streaming, deploying Spark to EC2?

• Go to https://spark.apache.org/docs/latest

• Navigate using tabs to the following areas in particular.

• Programming Guide > Quick Start, Spark Programming Guide, Spark Streaming, DataFrames and SQL, MLlib, GraphX, SparkR.

• Deploying > Overview, Submitting Applications, Spark Standalone, YARN, Amazon EC2.

• More > Configuration, Monitoring, Tuning Guide.

• Spark on Windows 7

• Why CPUs aren't getting any faster
• Hadoop: A brief History
• The State of Spark: And where we are going next
• https://blogs.apache.org/foundation/entry/the_apache_software_foundation_announces50

• Matei's Thesis
• Original (RDD) Paper

• http://blog.cloudera.com/blog/2015/07/how-to-do-data-quality-checks-using-apache-spark-dataframes/

• Distributed Systems for Fun and Profit
• Resilience Engineering: Learning to Embrace Failure
• Chaos Monkey

• PySpark Internals
• A deeper understanding of Spark's internals

• Tuning and Debugging in Apache Spark
• reduceByKey vs groupByKey
• Advanced Spark
• What's the difference between cache() and persist()
• Monitoring and Instrumentation

• A Tale of two clusters: Mesos vs. YARN

• https://plot.ly/ipython-notebooks/apache-spark/
• https://plot.ly/python/ipython-notebooks/
• https://plot.ly/python/matplotlib-to-plotly-tutorial/#6.1-Matplotlib-to-Plotly-conversion-basics

The word2vec tool takes a text corpus as input and produces the word vectors as output. It first constructs a vocabulary from the training text data and then learns vector representation of words. The resulting word vector file can be used as features in many natural language processing and machine learning applications.

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Distributed Representations of Sentences and Documents
• deeplearning4j tutorial (with applications)
• Modern Methods for Sentiment Analysis
• word2vec: an introduction

• Spark MLlib (Python, Scala)
• deeplearning4j (Java)
• gensim: word2vec and doc2vec (Python)

• Learning Spark: Lightning-Fast Big Data Analytics
  By Holden Karau, Andy Konwinski, Patrick Wendell, Matei Zaharia
  Publisher: O'Reilly Media, June 2014
  http://shop.oreilly.com/product/0636920028512.do
  Introduction to Spark APIs and underlying concepts.

• Spark Knowledge Base
  By Databricks, Vida Ha, Pat McDonough
  Publisher: Databricks
  http://databricks.gitbooks.io/databricks-spark-knowledge-base
  Spark tips, tricks, and recipes.

• Spark Reference Applications
  By Databricks, Vida Ha, Pat McDonough
  Publisher: Databricks
  http://databricks.gitbooks.io/databricks-spark-reference-applications
  Best practices for large-scale Spark application architecture. Topics include import, export, machine learning, streaming.

• Scala for the Impatient
  by Cay S. Horstmann
  Publisher: Addison-Wesley Professional, March 2012
  http://www.amazon.com/Scala-Impatient-Cay-S-Horstmann/dp/0321774094
  Concise, to the point, and contains good practical tips on using Scala.

• Spark Internals
  By Matei Zaharia (Databricks)
  https://www.youtube.com/watch?v=49Hr5xZyTEA

• Spark on YARN
  By Sandy Ryza (Cloudera)
  https://www.youtube.com/watch?v=N6pJhxCPe-Y

• Spark Programming
  By Pat McDonough (Databricks)
  https://www.youtube.com/watch?v=mHF3UPqLOL8

• Community
  https://spark.apache.org/community.html
  Spark's community page lists meetups, mailing-lists, and upcoming Spark conferences.

• Meetups
  http://spark.meetup.com/
  Spark has meetups in the Bay Area, NYC, Seattle, and most major cities around the world.

• Mailing Lists
  https://spark.apache.org/community.html
  The user mailing list covers issues and best practices around using Spark. The dev mailing list is for people who want to contribute to Spark.

This work by Jonathan Dinu is licensed under CC BY 4.0

• Share — copy and redistribute the material in any medium or format
• Adapt — remix, transform, and build upon the material for any purpose, even commercially.

The licensor cannot revoke these freedoms as long as you follow the license terms.

• Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
• No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation.

No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.