Active learning on the Hartmann test function using BoTorch's acquisition functions about botorch HOT 1 OPEN

I can't say I have a great understanding of the nuances of the active learning acquisition functions and why you see this behavior (or whether that is surprising), but let me share some thoughts:

• The metric you're using for evaluation here is the MSE, while qNegIntegratedPosteriorVariance targets, as the name suggests, the integrated posterior variance. It's not particularly surprising that the this may not lead to the best MSE model. It would be interesting to also compute the integrated posterior variance, w.r.t. which one would indeed expect better performance from using qNegIntegratedPosteriorVariance. More generally, it would be good to also include metrics that include a measure of the quality of the model's uncertainty quantification, e.g. the NLL of the observations under the model posterior.
• Does this replicate on other test functions? It's not uncommon to see one acquisition strategy show better performance one one problem and worse performance on another based on the nature of the underlying function. I would at least try a couple of other test functions to see whether this behavior is systematic or not.
• It looks like you're starting off at 100 initial points, which is quite a lot for a GP model on Hartmann already. You'll probably have a much easier time distinguishing performance if you reduce this substantially, since right now the difference in model performance will be quite small from the get-go.

from botorch.