In Active Learning literature, there is a heuristic called margin where we check the difference in probabilities between the first and second highest predicted class. If the margin is very low, it could indicate doubt.

Example:

cat: 0.9, dog: 0.05, elephant: 0.05 -> margin = 0.9 - 0.05 = 0.85 (high margin)

cat: 0.4, dog: 0.4, elephant: 0.2 -> margin = 0.4 - 0.4 = 0 (low margin)

https://twitter.com/KickItLikeShika/status/1507328167983394819?s=20&t=GIKUv7NsHW8e_lzJXW2vow

It makes sense to make it easy for folks googling' on "huggingface"/"torch"/"spaCy" to be able to find this.

When two models disagree something interesting might be happening. But that'll only happen if you have two models that are actually different.

What if you have one model that's better at accuracy and another one that's better at fairness.

Maybe these labels deserve more attention too.

If a machine learning model is very "confident" then the proba scores will have low entropy. The most uncertain outcome is a uniform distribution which would contain high entropy. Therefore, it could be sensible to add entropy as a reason for doubt.

Hey! First of all: this is a very cool project ;) I have been thinking about potential new "reasons" to doubt and I personally often look into predictions generated by a model whenever the data instance had missing values (and part of the model-pipeline imputes them)... So I wonder if it would be useful to have a FillNaNReason (or something similar) based, for example in the MissingIndicator transformer.

Real data is usually unbalanced but the long tail labels are also more likely to be wrong (whether predicted or human annotated). I don't know how much weight should be assigned to that though. But I figure it could make a Reason class on its own.

Some labels have a hierarchical or aggregative aspect that is predictive about label accuracy, e.g. in ImageNet there are 120 classes that are dog breeds which may be better or worse data compared to other groups of labels. This is more of a metadata thing.

And then there's straight up metadata, e.g. certain annotators may be very unreliable or if the date of annotation is known there may be a correlation that says most of the bad labeling happened at certain stage in the labeling process. But now we are moving into modeling the likelihood of a prediction being true or not.

Tbh it would be nice to have a meld of prodigy and doubtlab where we do active learning on whether the label was right or not (probably with tree-based method sticking to numbers and categories). And then fixing the labels while doing this.

And it would be nice if Prodigy had a more realistic demo for classification where there's hundreds or thousands of possible labels. Very easy to do something with these toy datasets - and when that's in the demo it usually means you have to fight the framework to do anything with a non-toy dataset.

Hey,

I'm trying to use doubtlab with my Keras model that does image classification. I have a model that is already trained and a test dataset. I want to find data from my test dataset that have a wrong classification or a good classification with low confidence.
Basically I'm running a very simple

model = tf.keras.models.load_model('data/results/SDH16K_GPU_WITHAUG/model.h5')
doubt = DoubtEnsemble(reason = WrongPredictionReason(model=model))
indices = doubt.get_indices(test_images, test_labels)

And get the following error:

105/105 [==============================] - 10s 50ms/step
File ~/code-project/MyoQuant-SDH-Train/.venv/lib/python3.8/site-packages/doubtlab/reason.py:232, in WrongPredictionReason.from_predict(pred, y, method)
    228         raise ValueError(
    229             f"Cannot use method={method} when y_true values aren't binary."
    230         )
    231 if method == "all":
--> 232     return (pred != y).astype(np.float16)
    233 if method == "fp":
    234     return ((y == 0) & (pred == 1)).astype(np.float16)
AttributeError: 'bool' object has no attribute 'astype'

I've tried wrapping my Keras Model as as Sci-kit classifier (using: https://www.adriangb.com/scikeras/stable/generated/scikeras.wrappers.KerasClassifier.html)
I get a "not fitted" error

sciKeras = KerasClassifier(model)
doubt = DoubtEnsemble(reason = WrongPredictionReason(model=sciKeras))

File ~/code-project/MyoQuant-SDH-Train/.venv/lib/python3.8/site-packages/scikeras/wrappers.py:993, in BaseWrapper._predict_raw(self, X, **kwargs)
    991 # check if fitted
    992 if not self.initialized_:
--> 993     raise NotFittedError(
    994         "Estimator needs to be fit before `predict` " "can be called"
    995     )
    996 # basic input checks
    997 X, _ = self._validate_data(X=X, y=None)

NotFittedError: Estimator needs to be fit before `predict` can be called

I guess DoubtLab is only for Scikit models for now, but I wondered if somebody tried something similar.

I have already started this one. Will push a PR once the conda installation option is available.

See: Adding doubtlab from PyPI to conda-forge channel.

@koaning As the primary maintainer of this repo, would you like to be listed as one of the maintainers of doubtlab on conda-forge channel? Please let me know, I will add your name as another maintainer of conda-forge/doubtlab-feedstock, once it is accepted.

If we assume classification with n > 2 classes, then the difference between the top two most confident predictions could also be used as a proxy for doubt. This could be the absolute difference or relative difference.

Issue

Environment details

Temporary fix

pip install "doubtlab==1.0.0"

More permanent fix

Pin doubtlab dependency to "doubtlab<2.0.0"

More more permanent fix

They've made some changes to their API

Let me know if you'd like me to make a PR

Thanks for a great package @koaning 😄

Although the apricot library isn't meant to find "bad" labels, it could help find the rows of data that "matter more". When starting out with only a few labels, this may be of great use.

Before moving on, it would be good to investigate the merits of this approach a bit more though.

Suppose there are 5 reasons for doubt, 4 of which overlap a lot. Then we may end up in a situation where we ignore a reason. That could be bad ... maybe it's worth exploring voting systems a bit to figure out alternative sorting methods.

There are a few ideas here. Right now we reward overlap and allow the user to use the predicates table to customize the sorting. But mayhaps we can go a step further. Maybe something like snorkel could play a role here.

Not 100% sure if it's worth the effort though, so it'd be good to benchmark first.

Suppose that y can contain NaN values if they aren't labeled. In that case, we may want to favor a subset of these NaN values. In particular: if they differ substantially from the already labeled datapoints.

The idea here is that we may be able to sample more diverse datapoints.

Maybe as a setting for WrongPredictionReason?

Per discussion in #28 (comment), I figured it'd be good to have some utility methods that may help in benchmarking all of these tools.

Hey! I was thinking if it would make sense to add two more reasons for regressions tasks, namely something like HighLeveragePointReason and HighStudentizedResidualReason.

Citing Wikipedia:

• Leverage is a measure of how far away the independent variable values of an observation are from those of the other observations. High-leverage points, if any, are outliers with respect to the independent variables (link)
• A studentized residual is the quotient resulting from the division of a residual by an estimate of its standard deviation. [...] This is an important technique in the detection of outliers. (link)

Maybe you should be able to add three as well?

Hi Vincent,

Nice library! As mentioned a while ago on Twitter I'm doing a review to understand and compare different approaches to find label errors.

I'm playing with the AG News dataset, which we know it contains a lot of errors from our own previous experiments with Rubrix (using the training loss and using cleanlab).

While playing with the different reasons, I'm having difficulties to understand the reasoning behind the MarginConfidenceReason. As far as I can tell, if the model is doubting the margin between the top two predicted labels should be small, and that could point to an ambiguous example and/or a label error. If I read the code and description correctly, MarginConfidenceReason is doing the opposite, so I'd love to know the reasoning behind this to make sure I'm not missing something.

For context, using the MarginConfidenceReason with the AG News training set yields almost the entire dataset (117788 examples for the default threshold of 0.2, and 112995 for threshold=0.5). I guess this could become useful when there's overlap with other reasons, but I want to make sure about the reasoning :-).

Hi, as stated in the title,
in reason.py line 260,
proba_dict = {classes[j]: v for j, v in enumerate(proba) if j != y[i]}
index j is compared to the class y[i]
suggested change:
proba_dict = {classes[j]: v for j, v in enumerate(proba) if classes[j] != y[i]}

I really like the current design with reasons just being function calls.

However, when working with large datasets or in use cases where you already have the predictions of a model, I wonder if you have thought about letting users to pass either a sklearn model or the pre-computed probas (for those Reasons where it make sense). For threshold-based reasons and large datasets this could save some time and compute, allow for faster iteration, and it would open up the possibility of using other models beyond sklearn.

I understand that the design wouldn't be as clean as it is right now, might cause miss-alignments if users don't send the correct shapes/positions, but I wonder if you have considered this (or any other way to pass pre-computed predictions).

Just to illustrate what I mean (sorry about the dirty-pseudo code):

class ProbaReason:

    def __init__(self, model=None, probas=None, max_proba=0.55):
        if not model or probas:
             print("You should at least pass a model or probas")
        self.model = model
        self.probas = probas
        self.max_proba = max_proba

    def __call__(self, X, y=None):
        probas = probas if self.probas else self.model.predict_proba(X)
        result = probas.max(axis=1) <= self.max_proba
        return result.astype(np.float16)

This seems to be a like-able method of dealing with text outside the realm of scikit-learn. But I prefer to delay this feature until I really understand the use-case. For anything related to entities we cannot use sklearn, but tags/classes should work fine as-is.