rubric={correctness:10,reasoning:10}

Investigate at least two of the following approaches to reduce memory usage while performing the EDA (e.g., value_counts).
Changing dtype of your data
Load just columns what we want
Loading in chunks
Dask
Discuss your observations.

• Very well-designed readme file.
• The report was perfect, and you have successfully done all the sections (More than what is needed!).
• Very well comparison in parts 3 and 4.
• Final “Challenges and Difficulties Faced” part was great!

In the textbox provided on Canvas for the Milestone 1 assignment include:

• The URL of your public project's repository
• The URL of your notebook for this milestone
• Copy & Paste Team Contract

Download the data from figshare to your local computer using the figshare API (you can make use of requests library).
Extract the zip file, again programmatically, similar to how we did it in class.
You can download the data and unzip it manually. But we learned about APIs, and so we can do it in a reproducible way with the requests library, similar to how we did it in class.

There are 5 files in the figshare repo. The one we want is: data.zip

SUBMISSION: Please put a link in canvas where TAs can find the following-

• Python 3 notebook, with the code for ML model in scikit-learn. (You can develop this on your existing jupyterHub in your EC2 instance, from milestone2)
• PySpark notebook, with the code for obtaining best hyperparameter settings. ( For this you have to use PySpark notebook in your EMR cluster )
  Screenshot from:
• Setup your EMR cluster (Task 1).
• Setup your browser , jupyter environment & connect to the master node (Task 2).
• Your S3 bucket showing model.joblib file. (From Task 3 Develop a ML model using scikit-learn)

https://docs.google.com/document/d/1Iz8kKGP5jHSW27Gl5XV01bRL-9WBlDdISTAE6AyMo1I/edit#

• Make sure the notebook runs from beginning to the end.
• Make sure notebook is well-documented and self-explanatory
• Discuss any challenges or difficulties you faced when dealing with this large data on your laptops.
• Briefly explain your approach to overcome the challenges or reasons why you were not able to overcome them.

rubric={correctness:15,reasoning:10}

Pick an approach to transfer the dataframe from python to R.
Parquet file
Feather file
Pandas exchange
Arrow exchange
Discuss why you chose this approach over others.

from slack:

2. When you pick a method to make your python dataframe available in R, make sure you give the reasoning behind it. This must be your team's reasoning/thoughts, and if you want, you can include some references to some articles. But the answer shouldn't be like "I picked it because I like it," "I picked this because you said it in class.", "This seems easy to me." - nothing that sort.

Use one of the following options to combine data CSVs into a single CSV.

Pandas
DASK
When combining the csv files make sure to add extra column called "model" that identifies the model (tip : you can get this column populated from the file name eg: for file name "SAM0-UNICON_daily_rainfall_NSW.csv", the model name is SAM0-UNICON)

Compare run times and memory usages of these options on different machines within your team, and summarize your observations in your milestone notebook.

Warning: Some of you might not be able to do it on your laptop. It's fine if you're unable to do it. Just make sure you check memory usage and discuss the reasons why you might not have been able to run this on your laptop.

• Write brief introduction of the project in the README.
• Create a folder called notebooks in the repository and create a notebook for this milestone in that folder.

Well done!

The results are fine, and you have done all the sections properly!