leave-class out validation about deepbgc HOT 1 CLOSED

Hi @pmobio,

Your approach should work. When you apply it to the original MiBIG used in the paper, can you reproduce our ROC results?

Our leave-class-out analysis was using an older version of deepbgc which didn’t have the train command yet, so it was a bit more complex. You can look into the dvc files here, although not sure how useful they are for the current deepbgc version: https://github.com/Merck/bgc-pipeline/tree/main/data/evaluation/lco-neg-10k

If you wanted to combine results across all BGC classes into one ROC curve, you will need to apply each model to its validation set, concatenate the prediction tsv files and then somehow generate the ROC from that using some of the deepbgc code.

I believe we also averaged the results across multiple models (multiple random seeds), not sure of that can be passed in the config json. You would also need to concatenate the predictions and create the ROC from that.

As for the memory, I can imagine it can use up a few GBs, but definitely not more than ~8, we were able to perform this on a laptop.

from deepbgc.