elsevier-ai-lab / bioblp Goto Github PK

Prepare BioKG data with benchmark triple identification. Ensure we mark triples that are part of benchmark datasets so we can remove them for model building. Output of task is sharing of the data.

• retrieve disease textual descriptions from MeSH

• compute LM-based encodings for disease entities

In this work, there is a lot of experimentation around KGE models for drug discovery.

Link to paper: paper

They are using the same BioKG as we do, which means that we should be able to replicate their results as long as we follow the same setup.

The script for the specific experiment is below.

#!/bin/bash
#SBATCH --job-name=bioblp-complex
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=6
#SBATCH --ntasks-per-node=1
#SBATCH --time=50:00:00
#SBATCH --mem=10G
#SBATCH --partition=gpu_shared
#SBATCH --gres=gpu:1

PROJ_FOLDER=BioBLP
OUT_FOLDER=models

# Copy data to scratch
cp -r $HOME/Projects/$PROJ_FOLDER $TMPDIR
cd $TMPDIR/$PROJ_FOLDER

source .venv/bin/activate

python -m bioblp.train \
--train_triples=data/ootb_biokg_triples/train_triples.csv \
--valid_triples=data/ootb_biokg_triples/val_triples.csv \
--test_triples=data/ootb_biokg_triples/test_triples.csv \
--dimension=464 \
--learning_rate=0.09 \
--num_epochs=600 \
--batch_size=128 \
--num_negatives=91 \
--log_wandb=True \
--loss_fn='NSSALoss' \
--notes="Replicating hyper-param experiment from Charlie paper"

# Keep files generated during job
RESULTS_FOLDER=$HOME/Projects/$PROJ_FOLDER-$OUT_FOLDER
mkdir -p $RESULTS_FOLDER
cp -r $TMPDIR/$PROJ_FOLDER/$OUT_FOLDER/* $RESULTS_FOLDER

I am currently running this experiment using the ootb-biokg-triples (out-of-the-box).

The purpose of this experiment is to pinpoint whether the removal of benchmark triples had such a huge effect in our performance, or whether there is an unidentified mistake on our side.

Similar to the paper here https://academic.oup.com/bioinformatics/article/36/2/603/5542390?login=true#192289302 , we want to be able to refer to scores of conventional approaches to the benchmark tasks. We need to index:

• What are the conventional models that do well, and specific works that SOTA the subtask (SVM / KronRLS)?
• What are the training and evaluation regimes?
• Does their data correspond to our version so we can point to their scores directly?
• What is the estimated complexity of implementing the method in our codebase?

As mentioned in above paper we have methods:

• DDR (Olayan et al., 2018)
• NRLMF (Hao et al., 2017)
• NRLMF (Liu et al., 2015)
• KRONRLS-MKL (Nascimento et al. 2016)
• COSINE (Lim et al., 2016)
• BLM-NII (Mei et al., 2013).

During training, the dictionaries mapping entities and relation to their IDs, are created independently for the training, validation, and test sets:

This means that incorrect embeddings are being used during validation and testing. Performance will thus be random although due to some ordering done when creating the dictionaries, in practice it's not completely random.