decile-team / distil Goto Github PK

Pull dataset loading functionality from CORDS to expand the breadth of datasets readily retrievable by DISTIL

SMI, SCMI, and SCG need to have self.model.eval() in their select() methods. Without it, the embedding computation will not necessarily be the same for similar points, which may cause performance degradation.

• Add the effective evaluation paper
• Add links to the SIMILAR and effective eval papers
• Remove the results right now from the main readme and rather link to the benchmark folders for each case. Each folder will have the detailed results for the corresponding case.

Add test folder that contains test suites (unit tests & others) for DISTIL

Add seed argument to stats.rv_discrete in L33 of badge.py

Reference Code: : https://github.com/laming-chen/fast-map-dpp

• SMI based AL (using query set)
• CSMI based AL (using a query + conditioning set)
• CG Based AL (using a conditioning set)

Add the benchmark results and notebooks corresponding to this paper: https://arxiv.org/pdf/2106.15324.pdf

Can you include examples of nlp applications, including the use of HuggingFace transformers?

Hello!

Is it possible to perform Active-Learning with a combination of SSL methods
such as Virtual Adversarial Training (VAT), Entropy Minimization (EntMin), etc?

I believe that this would be the major benefit of using DL for active learning.
Otherwise, one can use an easier model to train & tune after each iteration.

Do you also think Extreme Learning Machine could be useful as a one-shot
learning method to speed up the active-learning iterations with your library?

Implement a new condition for the data_train class that tests whether the testing accuracy has improved by a configurable amount within a configurable number of epochs. If this condition fails, then stop training.

Standardized way to train

The installation of submodlib (in requirements.txt) alone is taking 4min 49 seconds in Google Colab.

• Getting started
• CIFAR-10 example
• MNIST example
• Bio medical example
• Imbalance example
• Rare class example
• Redundancy example
• OOD data example

Hi Team,

I'm interest on utilizing https://github.com/decile-team/distil for active learning. My aim is to create a system where we select a subset of tasks from a larger pool and forward those representative samples to humans for manual review. Afterwards, it's crucial to distribute the decisions made on these representative tasks back to the subsets they were drawn from (or the subsets they represent). I'm exploring how to establish this reverse mapping from representative samples to subsets in order to effectively fan out our decisions.

Questions:

1. Does Distil offer an out-of-the-box feature for this?

2. If not, what would be the most effective approach to achieve it?

Though it says that distil coud be used with custom datasets there are no tutorials to support this claim and I have not been able to implement any kaggle datasets either. Please include instructions for if we want to use any dataset separate from the pre-defined

Summarize results in a table in the benchmark_notebooks folder and put plots/takeaways in each subfolder.

hello ，Could you provide some active semi-supervised learning algorithms? Besides, could you please pass my application about Decile_DISTIL_Dev group?

Weight Reset
Verbose
Maximum Accuracy

Hello authors,
It's great that you publish the source code, but the default hyper-parameters in config_cifar10_resnet_badge.json can not get similar accuracy in your project about badge strategy, so could you please share the hyper-params to produce the graph in README.

Hi, Thank you very much for the toolkit, I want to plot the experimental comparison result, but I don't konw how to plot the same effect as the paper, so can you provide the code for this plot? Thanks.

Benchmark notebooks require call to delete_checkpoints() at the end of train_one

If yes, please provide a tutorial

Consolidate examples, notebooks, and other expository python scripts into a tutorials folder to alleviate cluttering in base level of DISTIL

Is there any code for plotting the label efficiency?

Outdated based on the current GitHub repo code.

Add medical imaging benchmarks

Add SIMILAR notebooks to benchmark_notebooks folder and their results in the same manner as the effective evaluation notebooks