Datapreprocessing stuffs ? about tribuo HOT 4 CLOSED

We don't have an implementation of SMOTE, we tend to use example weighting to deal with imbalanced classes, and XGBoost has worked extremely well for that (we've deployed it in production on a severely imbalanced problem). If you want to contribute an implementation of SMOTE that would be pretty useful, but we're not likely to get to implement it ourselves any time soon.

Oversampling can be achieved using a user supplied function to generate the indices supplied to a DatasetView. This is similar to how we create the boostraps in BaggingTrainer or the resampling in AdaBoostTrainer. The AdaBoostTrainer creates a weighted bootstrap sample, which in practice oversamples difficult examples and undersamples easy examples (as defined by the AdaBoost.SAMME algorithm).

Tribuo comes with a built in text pipeline that can create a bag of words representation with term counting in TokenPipeline. If you want to convert that into TF-IDF then you could use the org.tribuo.transform infrastructure, write a custom transformer to calculate IDF, and then rescale all the term count features by the IDF (by applying the transformation). Writing custom transformers is a little tricky, but this one should be fairly straightforward. The transformation infrastructure is briefly discussed in the configuration tutorial.

from tribuo.

so in a nutsell you are basically saying that imbalanced data automatically managed under several algorithm we do no need to do much effort for it.that what you are saying ?.

from tribuo.

For Trainers which implement org.tribuo.WeightedExamples then I'd try setting appropriate example weights on the training dataset before building the model instead of performing oversampling. Oversampling loses some information and the reweighting doesn't. If the trainer you want doesn't support that then you can perform oversampling by generating your own indices and constructing a org.tribuo.dataset.DatasetView then use that as the training dataset.

from tribuo.

from tribuo.