the user should be able to do:

partial_dependence(feature_ids, predict_fn, classes=[1,2,4])

Could be a v2 request

instead of rendering rectangles, interpolate rbg points.

just to remove an extra step

i tried running lime for the sklearn breast_cancer dataset; as a result of the default kernel width, all the distances seemed huge, and in turn the sample weights were practically 0, so all the lime coeffients were 0.

2 options:

1. use feature scaling for the default kernel width
2. rescale the data before computing distances.

1. rename i.consider rename
2. restructure model module
3. rename model.processor

• color gradient bar
• better color choice, also lets keep this configurable as well through the function call

This is separate algorithm for interpreting trees and is useful only for the following types of models

• DecisionTreeRegressor
• DecisionTreeClassifier
• RandomForestRegressor
• RandomForestClassifier

This could possibly be an extension of the model interpretation interface. Extension because its model agnostic only wrt Tree/Ensemble based models.

https://www3.nd.edu/~rwilliam/stats3/Margins03.pdf

in requirements.txt, -e #egg=packagename will let you import packagename, but wont be recognizable to pip as package, so pip uninstall packagename doesnt work.

if users want to use sklearn==0.17, but our fork is 0.18, then there will be conflicts.

if authors update package, we need to merge those changes into our fork, creating potential issues.

Calling third party funcs behind our own function obscures the true function signature, which can cause errors, requiring us to maintain our pointer functions in accordance with the true, underlying function after an update.

The default PDP implementation makes use of weighted tree traversal for tree based model for improving performance.
Reference: http://scikit-learn.org/stable/modules/ensemble.html
"For each grid point a weighted tree traversal is performed: if a split node involves a ‘target’ feature, the corresponding left or right branch is followed, otherwise both branches are followed, each branch is weighted by the fraction of training samples that entered that branch. Finally, the partial dependence is given by a weighted average of all visited leaves. For tree ensembles the results of each individual tree are again averaged"

Currently we have

mean(y_hat) | X_i = x_j for all x_j in the perturbed space.

so if for instance:

sklearn returns this in deviation form:

Which do we think is more helpful?

This could be a useful feature to add.

So, R has some pretty neat implementations here.
Lets be pragmatic, on how we decide on extending model-interpretation. The goal is to make our library the de-facto implementation ppl decide to use for all forms of interpretation need.

make sure pyinterpret works on ec2 and platform.

This is a platform issue. Currently you cant have js visualizations on insights posts (plotly offline--same issue).

This pattern is enabled for FI but not for pdps:

f, axes = plt.subplots(2,2, figsize = (16, 16))

ax_dict = {
    'mlp':axes[0][0],
    'knn':axes[1][0],
    'reg':axes[0][1],
    'gb':axes[1][1]
}

interpreter = Interpretation()
interpreter.load_data(X_test, feature_names=data.feature_names)
for model_key in models:
    pyint_model = InMemoryModel(models[model_key].predict_proba, examples=X_train)
    ax = ax_dict[model_key]
    interpreter.feature_importance.plot_feature_importance(pyint_model, ax=ax)
    ax.set_title(model_key)

%matplotlib inline
from sklearn.datasets import load_boston, load_breast_cancer
from sklearn.ensemble import GradientBoostingRegressor, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression, LinearRegression
from pyinterpret.core.explanations import Interpretation
import pandas as pd
import numpy as np

classifier_data = load_breast_cancer()
classifier_X = classifier_data.data
classifier_y = classifier_data.target

classifier = GradientBoostingClassifier()
classifier.fit(classifier_X, classifier_y)

classifier_feature_id = [7]
classifier_feature_name = [classifier_data.feature_names[i] for i in classifier_feature_id]

classifier_feature_ids = [7, 23]
classifier_feature_names = [classifier_data.feature_names[i] for i in classifier_feature_ids]    

interpreter = Interpretation()
interpreter.load_data(classifier_X)
feature_ids = [classifier.feature_importances_.argsort()[-1]]
interpreter.partial_dependence.plot_partial_dependence(feature_ids, classifier.predict, with_variance=False)

fails:

TypeErrorTraceback (most recent call last)
<ipython-input-29-761529fb5028> in <module>()
      2 interpreter.load_data(classifier_X)
      3 feature_ids = [classifier.feature_importances_.argsort()[-1]]
----> 4 interpreter.partial_dependence.plot_partial_dependence(feature_ids, classifier.predict, with_variance=False)

/usr/local/lib/python2.7/dist-packages/pyinterpret-0.0.1-py2.7.egg/pyinterpret/core/global_interpretation/partial_dependence.pyc in plot_partial_dependence(self, feature_ids, predict_fn, class_id, grid, grid_resolution, grid_range, sample, sampling_strategy, n_samples, bin_count, samples_per_bin, with_variance)
    238                                       samples_per_bin=samples_per_bin)
    239 
--> 240         ax = self._plot_pdp_from_df(feature_ids, pdp, with_variance=with_variance)
    241         return ax
    242 

/usr/local/lib/python2.7/dist-packages/pyinterpret-0.0.1-py2.7.egg/pyinterpret/core/global_interpretation/partial_dependence.pyc in _plot_pdp_from_df(self, feature_ids, pdp, with_variance)
    291                 data = pdp.set_index(feature_name)
    292                 plane = data[mean_col]
--> 293                 plane.plot(ax=ax, color=color)
    294 
    295                 if with_variance:

/usr/local/lib/python2.7/dist-packages/pandas/tools/plotting.pyc in __call__(self, kind, ax, figsize, use_index, title, grid, legend, style, logx, logy, loglog, xticks, yticks, xlim, ylim, rot, fontsize, colormap, table, yerr, xerr, label, secondary_y, **kwds)
   3564                            colormap=colormap, table=table, yerr=yerr,
   3565                            xerr=xerr, label=label, secondary_y=secondary_y,
-> 3566                            **kwds)
   3567     __call__.__doc__ = plot_series.__doc__
   3568 

/usr/local/lib/python2.7/dist-packages/pandas/tools/plotting.pyc in plot_series(data, kind, ax, figsize, use_index, title, grid, legend, style, logx, logy, loglog, xticks, yticks, xlim, ylim, rot, fontsize, colormap, table, yerr, xerr, label, secondary_y, **kwds)
   2643                  yerr=yerr, xerr=xerr,
   2644                  label=label, secondary_y=secondary_y,
-> 2645                  **kwds)
   2646 
   2647 

/usr/local/lib/python2.7/dist-packages/pandas/tools/plotting.pyc in _plot(data, x, y, subplots, ax, kind, **kwds)
   2439         plot_obj = klass(data, subplots=subplots, ax=ax, kind=kind, **kwds)
   2440 
-> 2441     plot_obj.generate()
   2442     plot_obj.draw()
   2443     return plot_obj.result

/usr/local/lib/python2.7/dist-packages/pandas/tools/plotting.pyc in generate(self)
   1024     def generate(self):
   1025         self._args_adjust()
-> 1026         self._compute_plot_data()
   1027         self._setup_subplots()
   1028         self._make_plot()

/usr/local/lib/python2.7/dist-packages/pandas/tools/plotting.pyc in _compute_plot_data(self)
   1133         if is_empty:
   1134             raise TypeError('Empty {0!r}: no numeric data to '
-> 1135                             'plot'.format(numeric_data.__class__.__name__))
   1136 
   1137         self.data = numeric_data

TypeError: Empty 'DataFrame': no numeric data to plot

This issue is to extend the support to enable the ability to interpret deployed model globally using pdp

LIME talks about using the google inception model, lets be sure that we are able to do that successfully through our interface as well.
https://github.com/google/inception

so that travis will fail explicitly

pdp([feature1, (feature1,feature2), feature2], predict_fn)

sudo apt-get install libfreetype6-dev libxft-dev related to matplotlib

Because were distributing this function, tracebacks are kind of opaque. we should have more explicit logging and exception handling here to overcome this issue.

This issue is to compute feature importance in a model agnostic way as well. This will help in completing the story and adding more consistency.

This is to validate that the library is compatible wrt core functionalities for python 3 as well

This is to probably setup the binaries on the datascience private pypi server for distributing the binaries in a more organized way
https://pypi.datascience.com/#/

• Build simple Regression model using Keras, deploy it, and then interpret it globally and locally
• A text model using spaCy(A basic LDA), deploy it, and then interpret it locally.

1. with binary classification example - LR, RF/GBM, SVM
2. Regression example - linear RF/GBM,
3. Multi-class classification example
4. Multi-label classification example

Added SVM as well. Did I miss on anything else ?

1. Create a model in R and enable a REST API. There is an R package called plumber which could be useful. https://www.knowru.com/blog/how-create-restful-api-for-machine-learning-credit-model-in-r/ (this will reduce dependency on DS deploy). So, to be explicit the REST API runs locally, n thats ok, bcauz we are just enabling support for deployed model.
2. Using the above abstraction, we can see how we make it work for DS model_interpretation library

we infer the number of classes for classifiers by running model.predict on the dataset
say however you had data whereby the classifier only tended to output class 1 or class 2, though technically class 3 is possible
then if we perturbed data in such a way as to get a class 3 prediction back...
then wed have a bug

POTENTIAL SOLUTIONS:
1:
in the event that we see a different class in the perturbed data, just start the whole algorithm over with the updated information
2:
make those 3 model types separate functions, and for classifiers you specify the classes ( i dont like, what if the user messes up or doesnt know)
3:
we always have an additional null class. in the event that a new class is discovered (any new class), then it just inherits the effect of the null class, and the null class is displayed as "other classes"

This issue is to specifically look into improving the performance of LIME for local interpretation.

Im not convinced this will help, because regardless of defining c types and indexing etc we still need to rely on predict functions. Could be interesting though. I have a cythonized version that id like to build on (side project at home). Will PR here with benchmark results when ready.

Reference paper for the same: https://arxiv.org/pdf/1309.6392.pdf
This algorithm helps in capturing the variance affect across the range of covariates.

https://arxiv.org/abs/1612.08468

There could be potential performance issues that we might have to address here.

Possible alternatives:

1. datascience.ai.interpretation(:+1)
2. datascience.model.mai (_not sure what sub-module model stands for here_)
3. datascience.ml.mai (:+1)
4. datascience.mi

i mean that title is simply self explanatory

Too many if else is ugly and error prone and difficult to extend. When time permits lets address this.
This will help in reducing bugs. We should do this earlier than later because as we move forward we will forget about things.

current implementation:
https://datascienceinc.slack.com/files/aaron/F4T0D0QEA/pasted_image_at_2017_03_30_12_56_pm.png
h20 implementation:
https://www.dropbox.com/s/pxtdjkjzlp7lqpl/h20_categorical_variable.png?dl=0
Another alternative:
https://datascienceinc.slack.com/files/aaron/F4RJD2680/pasted_image_at_2017_03_30_12_56_pm.png

Possible to validate how it works with the current deploy. Ben could possible help us here.

This is to improve the performance of PDP using out of core parallelization