About structrue-enhanced self-attention about relphormer HOT 20 CLOSED

Hi, in this code version, we directly put strucural information as adjacency matrix in the self-attention module for simplicity.
It is easy to put structural encoder (an extra linear to encode the power of attention matrix).
Because the structural encoder is actually like a hyper-parameter and you can choose to use this module which is dependable on the actual knowledge graph you choose.
We will complete this part in the future version and also release the code version for relation prediction.

from relphormer.

you can find the implementation in the BertSelfAttention module (321 line.) in huggingface_relformer.py,

from relphormer.

Hi, thank you for your patient reply. I also want to ask why I can't enter BertSelfAttention in huggingface_relformer.py when I start from the main function for single-step debugging?

from relphormer.

Did you first pre-train the model for initialization and then load the pre-trained model?
The hyper-parameters of two stage are different and the second stage will import the huggingface_relformer.py.

from relphormer.

When performing pre-training or entity prediction, some files need to be downloaded, but it is very slow, so I downloaded it from https://huggingface.co/bert-base-uncased, am I right? The following is the hyperparameter setting for my running entity prediction.

from relphormer.

Yes, that is right. I think I find the different point.
You should run the "main.py" in the "Relphormer" directory instead of the same file in the "pre-train" directory.

from relphormer.

Sorry, I didn't present clearly, I just put the pre-trained model bert-base-uncased in the "pretrain" directory, this is run the "main.py" in the "Relphormer" directory.

from relphormer.

ok, can you check the program by putting the debug point in [from models.huggingface_relformer import [BertForMaskedLM]]
It imports the "models.huggingface_relformer" file.

from relphormer.

Sorry, I can only find the class BertForMaskedLM, but I can't find the code [from models.huggingface_relformer import [BertForMaskedLM]]

from relphormer.

Can you directly import the huggingface_relformer.py file? Because we rewrite the BertForMaskedLM class in this file.

from relphormer.

Do you mean to create a new file and try to import the huggingface_relformer.py file?

from relphormer.

Yes, you can try do that in the second training stage and it will also make sense.

from relphormer.

Single-step debugging of the main function to trainer.fit(lit_model, datamodule=data) will start training, unable to enter huggingface_relformer.py

from relphormer.

You can find the line which imports the model class.

from relphormer.

The hyperparameter setting of args.model_class is BertKGC. Will the model in huggingface_relformer.py be called during the execution of BertKGC?

from relphormer.

from relphormer.

Doesn't the attention_mask in the BertSelfAttention module (line 321) in huggingface_relformer.py actually build? I saw that attention_mask first came from the image below:

from relphormer.

Hi, we generate the attention mask for each center triple and you can find in the input of the module.

from relphormer.

Can you tell me exactly where in the input module?

from relphormer.

from relphormer.