The pre-train result is unsatisfactory with default parameter. about relphormer HOT 8 CLOSED

Hi, you should continue to run the next train step. The outputs in the pre-trainings stage are intermediate results.

from relphormer.

Sure, but the result is also unsatisfactory with default parameter.

In fine-tuning of WN18RR with default parameter, the result is as followed:

{'Test/hits1': 0.0001595405232929164,
'Test/hits10': 0.0001595405232929164,
'Test/hits20': 0.0006381620931716656,
'Test/hits3': 0.0001595405232929164,
'Test/mean_rank': 20385.440810465858,
'Test/mrr': 0.0003705603592570774}

In fine-tuning of umls with default parameter, the result is as followed:

{'Test/hits1': 0.01739788199697428,
'Test/hits10': 0.09379727685325265,
'Test/hits20': 0.17170953101361575,
'Test/hits3': 0.03555219364599092,
'Test/mean_rank': 58.33888048411498,
'Test/mrr': 0.05413661936632273}

from relphormer.

Hi, the result seems quite strange, do you correctly load the pre-trained weights?

from relphormer.

Hi, I only use the default parameter.

For the pre-training of umls:

nohup python -u main.py --gpus "0," --max_epochs=20  --num_workers=32 \
   --model_name_or_path  bert-base-uncased \
   --accumulate_grad_batches 1 \
   --model_class BertKGC \
   --batch_size 128 \
   --pretrain 1 \
   --bce 0 \
   --check_val_every_n_epoch 1 \
   --overwrite_cache \
   --data_dir /mine/Relphormer/dataset/umls \
   --eval_batch_size 256 \
   --max_seq_length 64 \
   --lr 1e-4 \
   >logs/pretrain_umls.log 2>&1 &

For the pre-training of WN18RR:

nohup python -u main.py --gpus "0," --max_epochs=15  --num_workers=32 \
   --model_name_or_path  bert-base-uncased \
   --accumulate_grad_batches 1 \
   --bce 0 \
   --model_class BertKGC \
   --batch_size 128 \
   --pretrain 1 \
   --check_val_every_n_epoch 1 \
   --data_dir /mine/Relphormer/dataset/WN18RR \
   --overwrite_cache \
   --eval_batch_size 256 \
   --precision 16 \
   --max_seq_length 32 \
   --lr 1e-4 \
   >logs/pretrain_wn18rr.log 2>&1 &

from relphormer.

There seems to be no problem for the parameters, perhaps you can check the model path again and re-train the whole model.

from relphormer.

Thanks for answer. I will try it again.

from relphormer.

Hi! we have updated the project and fixed the above running problem.
Sorry for the inconvenience.

from relphormer.

We have also released the pre-trained checkpoints, have a try.

from relphormer.