Hello,
I am trying to run the code on a graph I have built that is not included among the choices provided in the repo. The graph I am working with is bipartite with typed edges.

To run the code, I prepared the list of triplets to be split in train, valid and test sets.

Triplets are saved in .tsv files:

...
idx__518	id	2449.0
idx__519	id	2452.0
idx__523	id	2469.0
idx__531	id	2484.0
idx__532	id	2487.0
idx__533	id	2494.0
idx__549	id	2545.0
...

To run my dataset, I slightly modified the python scripts in the repo.

In preprocess_dataset.py I added the line datasets = ['mydata'] to read from the folder src/src_data/mydata. I was then able to run the script, which created and filled the folder data/mydata.

DATA_PATH: /content/ssl-relation-prediction/data
Preparing dataset mydata
2681 entities and 9 relations
creating filtering lists
Done processing!
1

In main.py, I modified the list of datasets by adding mydata so that the code wouldn't raise an exception.

Finally, I tried to run the code with the arguments specified in the readme:

! python src/main.py --dataset mydata --score_rel True --model ComplEx --rank 1000 --learning_rate 0.1 --batch_size 1000 --lmbda 0.05 --w_rel 4 --max_epochs 100

Unfortunately, at this point the code fails because there is no train.npy file in the data folder. I assume that the train.npy file should have been created by the preprocessing script, but for some reason that did not happen. The content of the data folder is the following:

total 1320
-rw-r--r-- 1 root root   40662 Dec  3 12:59 ent_id
-rw-r--r-- 1 root root      97 Dec  3 12:59 rel_id
-rw-r--r-- 1 root root   43023 Dec  3 12:59 test.tsv.pickle
-rw-r--r-- 1 root root 1083297 Dec  3 12:59 to_skip.pickle
-rw-r--r-- 1 root root  140727 Dec  3 12:59 train.tsv.pickle
-rw-r--r-- 1 root root   32391 Dec  3 12:59 valid.tsv.pickle

It's not clear to me how to run custom-made datasets from the readme. Could you help me with that?

Hi, thanks for your great contribution with this work :-) Do you plan to make the trained entity + relation embeddings for your SOTA models available for download?

Tara

Hi, I am trying your code an I think there is something wrong about the validation step in the training loop.

That is, consider the line at https://github.com/facebookresearch/ssl-relation-prediction/blob/main/src/engines.py#L197.
At this line, the value assigned to the variable split is actually the last value that was assigned in the previous loop for computing step-wise metrics over all the splits (L.192-195).
The last value is in fact "test".

So I think that at the end the model is wrongly validated against the test set instead of the validation set.

Hi Yihong, thank you so much for sharing this awesome repo!!

I tried to run the CP model; training seems to work, but when the validation starts, it crashes:

Evaluate the split train
Evaluating the rhs
Traceback (most recent call last):
  File "/home/jean/kg/src/main.py", line 25, in <module>
    main()
  File "/home/jean/kg/src/main.py", line 20, in main
    engine.episode()
  File "/home/jean/kg/src/engines.py", line 329, in episode
    self.validation_step(epoch=epoch, model=self.model)
  File "/home/jean/kg/src/engines.py", line 260, in validation_step
    res_s = self.dataset.eval(
  File "/home/jean/kg/src/datasets.py", line 243, in eval
    metrics = model.get_metric_ogb(
  File "/home/jean/kg/src/models/__init__.py", line 164, in get_metric_ogb
    cands = self.get_candidates(
TypeError: get_candidates() got an unexpected keyword argument 'indices'

Maybe the get_candidates(...) method of the CP class needs to be re-written?

Hi, where can I find the hyperparameters of the experimental results on each dataset in the Link Prediction Results section.

Hi,
I have noticed that in your experiments the flag --score_lhs is not enabled, and this flag includes the component $-\log P_\theta(s\mid p,o)$ into loss. In contrast, the 1vsAll objective includes this conditional likelihood, so it seems there is a discrepancy between the objective function in the paper (where there is a conditioning on the subjects) and the one used here.

Is it because you augment the data set with reciprocal triples? If so, is this equivalent to assuming that $P_\theta(S=s\mid R=p,O=o) = P_\theta(O=s\mid R=p^{-1},S=o)$, where $r^{-1}$ denotes the inverse relation?

Thank you

Hi! This is OGB Team.

We recently released https://github.com/snap-stanford/ogb/releases/tag/1.3.4 which improves the MRR calculation. This should not change your result if your model makes different predictive scores for different triplets, but would penalize those models that give the same predictive scores for different triplets (which is not ideal for link prediction).

Could you update your ogb package to 1.3.4 and run your model again just to confirm everything stays the same (the dataset stays the same. only evaluator changed)? Just one seed should be enough. Thanks!

Good evening. I have a question , regarding your code for the ogb wikidata. I've been working on graph neural networks lately, but because it's something new to me I'm having difficulties. Could I find somewhere documentation for the code and more specifically for the methods.Thanks in advance.

Hi, Yihong

I found a bug when i change the network of Complex and CP. I only add one linear layer for entity or relation embedding, than achieve unbelievable results:

Epoch: 0 TRAIN: {'MRR': 1.0, 'hits@[1,3,10]': [1.0, 1.0, 1.0]} VALID: {'MRR': 0.9929838180541992, 'hits@[1,3,10]': [0.9928429126739502, 0.993099570274353, 0.9931280612945557]} TEST: {'MRR': 0.9911285042762756, 'hits@[1,3,10]': [0.9909361600875854, 0.9912049174308777, 0.9914003610610962]}

You can reproduce the results by adding rel = self.fc(rel):

class ComplEx(KBCModel):
    def __init__(
            self, sizes: Tuple[int, int, int], rank: int,
            init_size: float = 1e-3
    ):
        super(ComplEx, self).__init__()
        self.fc = nn.Linear(2* rank, 2*rank)

    def score(self, x):
        rel = self.embeddings[1](x[:, 1])
        rel = self.fc(rel)

    def forward(self, x, score_rhs=True, score_rel=False, score_lhs=False):
        rel = self.embeddings[1](x[:, 1])
        rel = self.fc(rel)

I suspect there was a data leak, but I couldn't find the reason. Can you provide any ideas?

Looking forward to your reply.

Hi!thanks for your great work, I use your code to prepare dataset ogbl-biokg recently, i found the size of the test.pickle file and valid.pickle file are both 1.2GB,but train.pickle is 181MB, i'm not familiar with this dataset,so I don't know if it's normal