• NumPy

• Pandas

• Seaborn

• Matplotlib

• PySpark SQL

• PySpark ML

• Only need Anaconda with Python and Jupyter notebooks.

For this project, I was interested in predicting customer churn for Sparkify, a fictitious music streaming service.

The project entailed: - Loading and cleaning a small subset (128MB) of a larger available dataset

• Conducting exploratory data analysis to better understand the data and determine which features are useful for predicting churn.
• Feature Engineering, which involves creating features that will be used in the modeling process.
• Modeling with machine learning algorithms such as Logistic Regression, Random Forest, Gradient Boosted Trees, and Linear SVM.

To demonstrate my work in predicting churn, I have one Jupyternotebook with markdown cells were used throughout to explain my processes.

My Article could be found here Medium Blog Post By weighing F1 score and accuracy metrics, the best model was determined to be a Random Forest Classifier. The final model received an F1 rating and an Accuracy score of 0.75.

I'd like to thank Udacity for the project concept and workspace.

https://stackoverflow.com/questions/45946349/python-spark-cumulative-sum-by-group-using-dataframe https://stackoverflow.com/questions/35247168/spark-sql-row-number-partitionby-sort-desc https://sparkbyexamples.com/pyspark/pyspark-udf-user-defined-function/ https://machinelearningmastery.com/train-test-split-for-evaluating-machine-learning-algorithms/ https://sparkbyexamples.com/pyspark/pyspark-window-functions/ https://stackoverflow.com/questions/21702342/creating-a-new-column-based-on-if-elif-else-condition
https://stackoverflow.com/questions/46921465/extract-substring-from-text-in-a-pandas-dataframe-as-new-column
https://developers.whatismybrowser.com/useragents/explore/layout_engine_name/trident/
https://sparkbyexamples.com/pyspark/pyspark-when-otherwise/
https://stackoverflow.com/questions/52943627/convert-a-pandas-dataframe-to-a-pyspark-dataframe
https://stackoverflow.com/questions/29600673/how-to-delete-columns-in-pyspark-dataframe
https://stackoverflow.com/questions/48738354/having-troubles-joining-3-dataframes-pyspark
https://stackoverflow.com/questions/59886143/spark-dataframe-how-to-keep-only-latest-record-for-each-group-based-on-id-and
https://stackoverflow.com/questions/46956026/how-to-convert-column-with-string-type-to-int-form-in-pyspark-data-frame
https://medium.com/swlh/logistic-regression-with-pyspark-60295d41221
https://towardsdatascience.com/machine-learning-with-pyspark-and-mllib-solving-a-binary-classification-problem-96396065d2aa
https://spark.apache.org/docs/latest/ml-classification-regression.html#logistic-regression
https://stackoverflow.com/questions/60772315/how-to-evaluate-a-classifier-with-apache-spark-2-4-5-and-pyspark-python
https://spark.apache.org/docs/2.2.0/ml-classification-regression.html
https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html
https://stackoverflow.com/questions/32565829/simple-way-to-measure-cell-execution-time-in-ipython-notebook
https://www.silect.is/blog/random-forest-models-in-spark-ml/
https://stackoverflow.com/questions/75440/how-do-i-get-the-string-with-name-of-a-class