The IFA tool is used to find if a feature has incremental predictive power over an existing model. While literature discusses looking at forecast error (i.e.
The IFA Tool does not assume any relationship between the contesting feature and the performance. Rather, we segment the feature using an arbitrary segmentor and observe the model error in local "neighborhoods" defined by the segmentor and use visual judgement to determine what pre-processing, if any, the feature will require.
pip install ak-ifa
requirements are listed in the requirements.txt
file
Given a dataframe with at least:
- 1 column of labels {0,1} or any values in the range (0,1), called 'y' in the example
- 1 column of predictions as probabilities in the range (0,1), called 'p' in the example
- 1 column of a candidate continuous feature, called 'x' in the example
from ifa.api import *
result, _ = analyze_feature(df, x_col='x', p_col='p', y_col='y', plot=True)
Will plot a graphical test for the added information of the feature 'x' over the prediction 'p'.
To ease learning, we have provided a demo module with some helper functions
import numpy as np
import pandas as pd
from ifa.api import *
from ifa.demo import model_samples, fit_logistic_regression
from ifa.math import sigmoid
# assume linear coefficients of a logistic regression model
beta = np.array([1, -1, 2, -2])
# create model samples based of coefficients (see math below)
X, y, _ = model_samples(10000, beta, seed=42)
# fit a model on part of the features
X_partial = X[:,:3]
beta_hat_partial = fit_logistic_regression(X_partial, y)
print(beta_hat_partial) # [ 0.78434972 -0.79164423 1.61375143], values might depend on RandomState's impl.
y_hat_partial = sigmoid(X_partial.dot(beta_hat_partial))
df_partial = pd.DataFrame({
'x':X[:,3], # the 'missing' feature
'y': y,
'y_hat': y_hat_partial
})
result, _ = analyze_feature(df_partial, x_col='x', p_col='y_hat', y_col='y', plot=True)
import numpy as np
import pandas as pd
from ifa.api import *
from ifa.demo import model_samples, fit_logistic_regression
from ifa.math import sigmoid
# assume linear coefficients of a logistic regression model
beta = np.array([1, -1, 2, -2])
# create model samples based of coefficients (see math below)
n = 10000
X, y, _ = model_samples(n, beta, seed=42)
# fit a model on all features
beta_hat = fit_logistic_regression(X, y)
y_hat = sigmoid(X.dot(beta_hat))
df = pd.DataFrame({
'x':np.random.uniform(-1,1,n), # the 'missing' feature, this time with no information
'y': y,
'y_hat': y_hat
})
result, _ = analyze_feature(df, x_col='x', p_col='y_hat', y_col='y', plot=True)
When using the CT to find sloping features for a production model we should follow these steps:
- Use the tool to determine if the feature can improve the model.
- Decide, based on quantitative thresholds and qualitative considerations if the feature should be added to the model.
- Pre-process the feature. repeat steps 1, 2 if necessary.
- Add the processed feature to the model.
In this doc, we will cover steps 1, 4. we will also cover some mathematical properties of CT through simulation.
- CT's Feature analysis can be used with any probability generating model.
- CT can operate with any real or categorical feature.
- The suggested correction can also be used for any probability generating model, but would fit very nicely and linearly with Logistic Models.
Let:
Let
The set
For each numpy.qcut
Over a dataset that is defined by the below tuple:
Define
The rationale behind the feature analysis assumes:
- Extra features should tested 'above' or 'given' the model.
- Although we use logistic regression which is GLM, we don't assume any behavior of the feature
- Rather, we check if there's a constant that we can add to the model in different neighborhoods of
$x$ that will minmize it's Cross Entropy within that neighborhood.
Let
Let
The Cross Entropy in neighborhood
For each neighborhood we check if there exists a bias
In words:
It's easy to see that
Define
Plot the pairs:
The procedure is as follows:
- Draw some intercept
$$\beta_0\in\mathbb{R}$$ - Draw a weight vector
$$\mathbf{\beta} \in [-1,1]^p$$ - Assign
$$\text{results} = []$$ (an empty list)
- For k in 1,2,3,...,N repeat:
- draw
$\mathbf{x}_k$ from$[-1,1]^p$ - assign
$\mu_k \leftarrow \beta_0 + \mathbf{x}_k^T\mathbf{\beta}$ - assign
$\pi_k \leftarrow \frac{e^{\mu_k}}{1+e^{\mu_k}}$ - draw
$y_k$ from$\text{Ber}(\pi_k)$ (either 0 or 1) - assign
$\text{results}[k] \leftarrow (y_k, p_k, \mu_k, \mathbf{x}_k)$
Learn a partial model
Once we agree with the definition of the feature (we might want to perfrom some pre-processing) we would like to update the model's separation hyperplane with an additional coordinate. To do so we keep the separation hyperplane constant and only allow a new slope and bias to be learned.
Let:
Given a dataset that is selected for the procedure:
Define:
Learn