blackhc / batchbald_redux Goto Github PK

Hi @BlackHC,

Thanks a lot for creating this amazing library. I have a question regarding the (active) training loop in the example here:

At the end of each iteration in the while loop we acquire the new data via

active_learning_data.acquire(candidate_batch.indices)

That is, the active_learning_data.training_dataset will have new instances. Here it comes my doubt,

how the acquired instances are then included in the train_loader?

The train_loader is defined outside the while loop and does not change throughout training. It seems to me that the model sees always the same data in the training loop

# Train
for data, target in tqdm(train_loader, desc="Training", leave=False):
    data = data.to(device=device)
    target = target.to(device=device)
     ...

Shouldn't it be able to train on the "new" (extended) training set, i.e. shouldn't the train_loader be redefined at each iteration?

To provide a bit of context: I am trying to write a model leveraging both batchbald_redux and baal. Out of curiosity, can you comment on the differences between the two libraries and why you chose to reimplement some things (e.g., the ActiveLearningData?). I am trying to combine them and get the best of each one. Also, I have seen that they included an implementation of BatchBald taken from the code you provided originally with the paper, is there an advantage on using that implementation vs this one? - Well, I guess these are a bonus questions, feel free to ignore :)

Thanks a lot in advance for your attention.

Best,
Pietro

I see that there is a provided MNIST image classification example in the repo. I was wondering if we could extend this to cover object detection tasks.

Hi,
I am inspecting the fields of application of BatchBald. I am using the get_batchbald_batch function from https://github.com/BlackHC/batchbald_redux/blob/master/01_batchbald.ipynb

Here is an example: we have 3 samples with 2 mc inferences for each with 4 classes. The first two examples are totally identical, while the third is completely different yet has slightly lower BALD score (due to smaller fluctuation of probabilities across mc runs):

log_probs_N_K_C = torch.Tensor([
[[0.1, 0.2, 0.3, 0.4], [0.15, 0.15, 0.3, 0.4]],
[[0.1, 0.2, 0.3, 0.4], [0.15, 0.15, 0.3, 0.4]],
[[0.4, 0.3, 0.2, 0.1], [0.4, 0.3, 0.16, 0.14]],
]).log()

I hoped BatchBald would be useful in such a case since it queries diverse instances for the batch. Yet, it queries two duplicate examples:

get_batchbald_batch(log_probs_N_K_C, batch_size=2, num_samples=3) outputs [0, 1].

I wonder whether this is the expected behavior of the algorithm or some bugs in the code may be present.
Thank you for your attention!

Hey! Awesome piece of work. I've just been going through the code and I'm curious what these two statements are for

It seems to me as if they won't actually change which candidate is selected as the maximum, as it's adding a float to all elements of the vector scores_N

Even though the name makes it obvious that probabilities are expected by most functions, it would be better to change the API to use log probabilities.

This would mean that log_softmax outputs can be passed directly without having to call .exp_().

It's too easy to forget that and leads to NaN that take time to debug.