ericfourrier / auto-clean Goto Github PK

View Code? Open in Web Editor NEW

7.0 7.0 5.0 3.71 MB

Provide helpers in python (pandas and numpy) to perform automated cleaning and pre-processing steps.

License: MIT License

Python 100.00%

auto-clean's People

Contributors

Stargazers

Watchers

Forkers

bruledamien paurichardson wade1990 lgtm-migrator

auto-clean's Issues

Implement test for DataExploration.detect_other_na()

Build first automatic Imputer ( Ensemble model)

Add the possibility to make a copy for DataExploration class with inplace argument

Test if None is count like np.nan in pandas

Correlation plot in not enough example in give me some credit

Replace staticmethod in NaImputer

Rethink strong outliers based only on strong_cutoff to take care of lognormal distribution

Add first draft of OutlierDetection Class

Add a first draft of README.md

Create new infos printer for DataExploration

Add method to test colname that are possible int factor

Put a special treatment for datetime col in fillna_serie

Create Multiplot density for is_na_col

Update sample constant_col and detect key

Add a detector for not pandas np.nan like 'N/A', 'Missing' ...

/home/travis/miniconda/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

Implement corrplotna

implementation of a correlation plot of dummy variables is_na_col

Add method to put all string to lowercase for str columns

Na imputation in test train or in pandas dataframe

From Olivier Grinsel tutorial

When doing imputation in pandas, prior to computing the train test split we use data from the test to improve the accuracy of the median value that we impute on the training set. This is actually cheating. To avoid this we should compute the median of the features on the training fold and use that median value to do the imputation both on the training and validation fold for a given CV split.

View the update logs.

We can think about shannon divergence criteria
mean and std if shape = gaussian