Details on how to reproduce experiments about pykeen HOT 2 CLOSED

Hi @LFGMUW,

I tried following the documentation on how to train and evaluate embedding models in pykeen but after the OpenBioLink Dataset caused CUDA oom issues (partly because for most models slicing is not implemented or can I adapt any parameters to fit it?

Please consider that OpenBioLink is a comparably large dataset. To avoid CUDA out of memory exceptions, you can reduce the embedding dimension of your model and make use of the automatic memory optimization, i.e. pipeline(model_kwargs=dict(automatic_memory_optimization=True, ...), ...). Please note that if your model uses batch normalization, the automatic_memory_optimization cannot be used since it relies on sub-batching whereas batch normalization requires the full batch when applied.

sidenote: how are the OBLF1 and F2 supposed to me used?

Openbiolinkf1 and Openbiolinkf2 are subsets of OpenBioLink that we created, which we will upload soon.

from pykeen.pipeline import pipeline
pipeline_result = pipeline(
dataset='WN18RR',
model='RotatE',
model_kwargs=dict(
embedding_dim=500,
#automatic_memory_optimization=True,
)

We performed a reproducibility study and integrated the corresponding configurations. The configuration for RotatE on WN18RR can be found at https://github.com/pykeen/pykeen/blob/master/src/pykeen/experiments/rotate/sun2019_rotate_wn18rr.json
The best RotatE-WN18RR configuration that we found is available at https://github.com/pykeen/benchmarking/blob/master/ablation/results/rotate/wn18rr/random/adam/2020-04-25-19-04_217bcf38-2101-461b-9593-b133b4201e6a/0000_wn18rr_rotate/best_pipeline/pipeline_config.json

Please let us know if you need further help :)

from pykeen.

Hi @mali-git,
Thank you for replying to my questions.

Please consider that OpenBioLink is a comparably large dataset. To avoid CUDA out of memory exceptions, you can reduce the embedding dimension of your model and make use of the automatic memory optimization

Yes, but I thought since it is built-in there would perhaps be a way of training, I did use using the memory optimization parameter as well and ran it with dimension 50, making it much smaller does not seem promising for the learned embeddings.

Please note that if your model uses batch normalization, the automatic_memory_optimization cannot be used since it relies on subbatching whereas batch normalization requires the full batch when applied.

Yes, thank you. I was hoping there was a way around the issue of memory constrains seeing that you published results to the models.
I did see the experiment configs, they where the basis of my attempt of recreating. I was surprised how many orders my metric results differed. But thank you.

from pykeen.