viniciusmsousa / pyspark-ds-toolbox Goto Github PK

https://livebook.manning.com/book/data-analysis-with-python-and-pyspark/chapter-14/v-14/

Add a prediction function that replicates the preparation pipeline of the training function.

I think it is only used to test the association class, loading the iris dataset.

Add life cycle badges to the functions.

Based on tidy verse: https://lifecycle.r-lib.org/articles/stages.html

Add report on test coverage.

When adding '{os: macOS-latest, python: 3.7}' to package-tests.yaml it fails saying that pandas was not found.

The main one if the parameter that receives the DFs. It changes in some functions. Choose one and use it in all functions.

Update the Association Class (stats.association) to work with pyspark.pandas dataframes, since Koalas was fully integrated im pyspark.

It would be nice to have a feature selection funcitonality

• Check: https://github.com/smazzanti/mrmr
• Check https://github.com/jvicentem/big_mrmr
• Decide wheter to implement or not this functionality
• Implement or not

Stop reading/loading from the internet the test files.

Add example os the usage of the functionalities. My ideia is to create a folder called examples and have notebooks inside this folder.

• Add examples folder;
• List the functions that need examples to be added;
• Create a PR that adds such examples.

This issue then will be managed from that PR

Check here to see lines that are not tested and test them.

Add an implementation of information value compuation.

• Set None as default value (so it can be ommited);
• Add an if else statement so that features is only the non None list.

Design a vif functionality that works well within pipelines.