zjunlp / mkgformer Goto Github PK

Hi, thanks for your contributions. when I run MNER task, i.e., twitter17, the final result is a little different from the paper. I finally get f1 86.54 by running four times but it should be 87.49. Does the torch version or other factors influence the final results?

My env is Tsela K40, Cuda 10.1 Torch 1.7.0

from models.unimo_model import UnimoCRFModel
I am getting no module named models.unimo_model from where do I install this package and which package it is

粗略统计一下txt部分的：train.txt中有12247条数据，其中就有7751个None的无关系数据；test.txt中有1614条数据，其中就有974个None的无关系数据。大部分文本关系都是None，这样真的能学到实体关系信息吗？而且test的None数据也占到一半多，得到的分数是否不准确啊？

还要一点就是我发现子图中：RCNN抽取objects是比visual grounding抽取objects效果好一点，但他们的objects其实都没有准确抽到实体，大部分都只是重复一下原图的部分子图片。想问您一下这是什么情况啊？

您好，我想要与您的方法进行对比。请问一下，这个checkpoints的预训练方法是什么？

最近在跑代码的时候出现下面的错误
Traceback (most recent call last):
File "main.py", line 169, in
main()
File "main.py", line 151, in main
trainer.fit(lit_model, datamodule=data)
File "/home/E/anaconda3/envs/MKGC/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
self._run(model)
File "/home/E/anaconda3/envs/MKGC/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 756, in _run
self.dispatch()
File "/home/E/anaconda3/envs/MKGC/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 797, in dispatch
self.accelerator.start_training(self)
File "/home/E/anaconda3/envs/MKGC/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
self.training_type_plugin.start_training(trainer)
File "/home/E/anaconda3/envs/MKGC/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
self._results = trainer.run_stage()
File "/home/E/anaconda3/envs/MKGC/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 807, in run_stage
return self.run_train()
File "/home/E/anaconda3/envs/MKGC/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 842, in run_train
self.run_sanity_check(self.lightning_module)
File "/home/E/anaconda3/envs/MKGC/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1107, in run_sanity_check
self.run_evaluation()
File "/home/E/anaconda3/envs/MKGC/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 962, in run_evaluation
output = self.evaluation_loop.evaluation_step(batch, batch_idx, dataloader_idx)
File "/home/E/anaconda3/envs/MKGC/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 174, in evaluation_step
output = self.trainer.accelerator.validation_step(args)
File "/home/E/anaconda3/envs/MKGC/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 226, in validation_step
return self.training_type_plugin.validation_step(*args)
File "/home/E/anaconda3/envs/MKGC/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in validation_step
return self.lightning_module.validation_step(*args, **kwargs)
File "/home/E/python/MKGformer-main/MKG/lit_models/transformer.py", line 105, in validation_step
result = self._eval(batch, batch_idx)
File "/home/E/python/MKGformer-main/MKG/lit_models/transformer.py", line 96, in _eval
assert logits.shape == labels.shape
AssertionError
出现问题是在pretrain步骤，请问出现问题的原因可能是什么。我该如何解决，谢谢！

您好，运行MKG部分代码的时候找不到 self.num_training_steps, 请问这个 self.num_training_steps 在哪里定义的，谢谢百忙之中抽空解答~

File "/home/slyang/MKG/lit_models/transformer.py", line 157, in configure_optimizers
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=self.num_training_steps * self.args.warm_up_radio, num_training_steps=self.num_training_steps)
File "/home/slyang/.conda/envs/kg_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1208, in getattr
type(self).name, name))
AttributeError: 'TransformerLitModel' object has no attribute 'num_training_steps'

Dear all,

We found a severe bug in the data processing code for FB15K-237-IMG, which leads to an unfair performance comparison. We are very sorry for that.

The bug was caused by a data processing function that wrongly feeds the images of head entities similar to the images of tail entities (those images should be filtered for a fair comparison). Due to poor version control, we mistakenly used the buggy version to obtain the performance.

The bug has been fixed now. We have updated the performance in arxiv and released the checkpoints (The model trained with/without the severe bug) for reproducibility.

The correct fully-supervised performance (without carefully hyper-parameter tuning) is hit@1 0.256 hit@3 0.367 hit@10 0.504 MR 221, which is still better than previous multimodal baselines and does not affect the paper's conclusion.

Sincerely,
All authors

你好，请问这是哪里的问题

感谢您的解答，还有一个问题不太清楚。我运行了MRE任务，为什么设置了随机种子，但每次运行结果不一样，而且代码的运行结果与论文中给出的结果不一致，F1的真实结果比论文中下降了1%多个点，准确率的真实值在90%左右，而论文中是85%左右。

Traceback (most recent call last):
File "run.py", line 11, in
from models.unimo_model import UnimoCRFModel
ModuleNotFoundError: No module named 'models.unimo_model'
I understand that the models.unimo_model is located in the same-level folder as run.py, and you are still encountering the issue even after following the instructions in the readme file and adding the parent folder path using sys.path.append.

你们好，再次打扰一下哈。关于Visual Grounding,想请教一些问题，对于关系抽取任务
1）在第一步解析名词短语时，我看数据集还给出了头实体和尾实体，这个是也会作为名词短语，用于输入到visual grounding工具中吗？
2）One stage visual grounding一共提供了两个checkpoint，我看这两个数据集中，实体一般比较粗，例如人只有man这种粒度的，而MRE数据集中，其一般为命名实体。是否在第一步parser之后，直接用parser出的名词短语直接输入到visual grounding工具中，还是要进行一些处理来识别出所抽取名词短语的类别呢？

非常感谢！

The folder FB15K-images and wn-18 images in the path [MKGformer]/dataset/ contain no images. Can you please provide the image data

请问作者，您的模型中的uniMomodel的vision hidden state是没有做任何处理的，这个vision hidden state里面包含的信息是否是可以和text hidden state替换的，与text hidden state是等价的，者两者使用起来没有什么区别；或者不能等价，是考虑将uniMomodel作为多模态知识图谱的编码器，可以在其他下游任务上面使用呢？

请问在运行这两个脚本文件时，我是使用Git运行的，出现的这个错误是什么原因呢

Is it possible to predict tail entity T (h,r,?) only from the head entity H and relation R in MKG task? Is it possible to perform (? ,r,t) link prediction task? Since I did not see the specific description of the prediction header entity in detail in the paper, I really hope you can answer. Thank you!

作者您好，我认为您的工作做的非常有意义，所以最近在利用您所给出的FB15k-237和其IMG数据集进行论文实验结果的复现，lr和batzhsize均按照脚本文件给出的值设定，但是实验结果并不理想，和论文给出的值差别较大。想请问一下如何才能获得论文中的实验结果，谢谢。

Hi,你们好，请问link prediction部分，看论文好像说对Bert的词表进行了扩充？这个是不是应该上传相应vocab或者config文件？我看代码好像就这里提到了，如果是我没注意到还请告诉我一下，感谢~

vision_config = CLIPConfig.from_pretrained('/home/lilei/package/clip-vit-base-patch32').vision_config
    text_config = BertConfig.from_pretrained('/home/lilei/package/bert-base-uncased')
    bert = BertModel.from_pretrained('/home/lilei/package/bert-base-uncased')

AttributeError: module 'models' has no attribute 'RobertaUseLabelWord'

ERROR: Could not find a version that satisfies the requirement torchvision==0.8.2 (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3, 0.10.0, 0.11.2, 0.11.3, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1)
ERROR: No matching distribution found for torchvision==0.8.2

I am getting this error while installing torchvision==0.8.2 which version I should use to run this code

为什么要把他们三者分别作为一种feature呢

您好，我按照requirements.txt进行安装环境后遇到以下几个主要问题：
在Python3.8环境中：
1.dataclasses没有0.8版本，只有0.6版本，Pypi中说是因为在Python3.8中已经被整合到Python中了。
我选择安装dataclasses==0.6
2.torchvision与torch版本不匹配，torchvision==0.8.2只匹配torch==1.7.1
我选择安装torch1.7.1
3.ImportError: cannot import name 'get_num_classes' from 'torchmetrics.utilities.data'
这个可能与我安装中自作主张有关
但是如果选择的是python3.6环境的话，安装的scikit-learn版本无法超过1.0
请问以上问题您当时有没有遇到过，请问您能不能提供一些解决办法？

按照readme中显示流程，对fb15k-237预训练(pretrain_fb15k-237-image.sh)，在第五个epoch就达到了hit@1=0.99，用此模型进行训练(fb15k-237-image.sh)，得到的结果hit10为0.504，请问是否是我的训练参数没有调整正确呢，或者是训练过程不对呢

您好，可以上传这篇论文的数据集到百度网盘吗？

Hi authors, thanks for your paper as well as your code. However, when I download the entire dataset from your baidu network disk and run the MKG task with the script you provided, the results are almost always 0. I wonder if something is wrong here.

Hi there, nice work. There is a detail I am not very clear. In few-shot learning of MRE, did you just randomly sample K examples for each relation type to construct the few-shot training dataset and train MKGformer using the constructed dataset？ I noticed that in FL-MSRE, some few-shot techniques are used, did you use similar techniques in this paper?

Hi, there. Thanks for your work. I meet a bug when run MNER run.py at passing CRF. The detail is
CUDA error: device-side assert triggered
File "/home/ein.cf/MNER/code/MKGformer/MNER/models/unimo_model.py", line 53, in forward
loss = -1 * self.crf(emissions, labels, mask=attention_mask.byte(), reduction='mean') # 去掉CLS
File "/home/ein.cf/MNER/code/MKGformer/MNER/modules/train.py", line 279, in _step
output = self.model(input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids, labels=labels, images=images, aux_imgs=aux_imgs, rcnn_imgs=rcnn_imgs)
File "/home/ein.cf/MNER/code/MKGformer/MNER/modules/train.py", line 76, in train
attention_mask, labels, logits, loss = self._step(batch, mode="train")
File "/home/ein.cf/MNER/code/MKGformer/MNER/run.py", line 141, in main
trainer.train(clip_model_dict, text_model_dict)
File "/home/ein.cf/MNER/code/MKGformer/MNER/run.py", line 146, in
main()

Did you met this before? The general solution of this cuda issue is to make sure the range of tuple. But I can not print it and don't know the reasonable range

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/data01/home//miniconda3/envs/idea/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/data01/home/miniconda3/envs/idea/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/data01/home//MKGformer/MKG/data/processor.py", line 554, in encode_lines
enc_lines.append(json.dumps(self.convert_examples_to_features(example=eval(line))))
File "", line 1
{"guid": "train-0", "text_a": "[MASK]", "text_b": "film[PAD]", "text_c": "[UNK] Austin Powers: International Man of Mystery is a 1997 American action comedy film and the first installment of the Austin Powers series. It was directed by Jay Roach and written by Mike Myers, who also starred as both Austin Powers and the antagonist Dr. Evil., Powers' arch-enemy. The film co-stars Elizabeth Hurley as Vanessa Kensington, Robert Wagner as Number 2, Seth Green as Scott Evil, and Michael York as Basil Exposition. Will Ferrell, Mimi Rogers, Carrie Fisher, Tom Arnold, Rob Lowe, Christian Slater, Cheri Oteri, Neil Mullarkey and Burt Bacharach made cameo appearances alongside an uncredited cameo by MADtv star Michael McDonald, among many others. The film is a parody of the James Bond films as well as other parody films that also spoof James Bond and other 1960s spy films. The film, which cost US$16.5 million, opened on May 2, 1997
^
SyntaxError: EOL while scanning string literal
"""

您好，想请问一下，在运行MKG任务中bash scripts/pretrain_fb15k-237-image.sh这条指令时有如下报错，是什么原因呢。模型是在huggingface网站上下载后存放在本地的
(mkg) lab401@gpu-1:~/MKGformer-main/MKG$ bash scripts/pretrain_fb15k-237-image.sh
Namespace(accelerator=None, accumulate_grad_batches=1, amp_backend='native', amp_level='O2', auto_lr_find=False, auto_scale_batch_size=False, auto_select_gpus=False, batch_size=64, bce=0, benchmark=False, check_val_every_n_epoch=1, checkpoint=None, checkpoint_callback=True, chunk='', data_class='KGC', data_dir='dataset/FB15k-237', dataset='./dataset/NELL', default_root_dir=None, deterministic=False, distributed_backend=None, eval_batch_size=64, fast_dev_run=False, flush_logs_every_n_steps=100, gpus='0,', gradient_clip_algorithm='norm', gradient_clip_val=0.0, label_smoothing=0.1, limit_predict_batches=1.0, limit_test_batches=1.0, limit_train_batches=1.0, limit_val_batches=1.0, litmodel_class='TransformerLitModel', log_every_n_steps=50, log_gpu_memory=None, logger=True, lr=0.0005, max_epochs=15, max_seq_length=64, max_steps=None, max_time=None, min_epochs=None, min_steps=None, model_class='UnimoKGC', model_name_or_path='/home/lab401/MKGformer-main/clip-vit-base-patch32', move_metrics_to_cpu=False, multiple_trainloader_mode='max_size_cycle', num_nodes=1, num_processes=1, num_sanity_val_steps=2, num_workers=4, optimizer='AdamW', overfit_batches=0.0, overwrite_cache=True, plugins=None, precision=32, prepare_data_per_node=True, pretrain=1, process_position=0, profiler=None, progress_bar_refresh_rate=None, reload_dataloaders_every_epoch=False, replace_sampler_ddp=True, resume_from_checkpoint=None, seed=7, stochastic_weight_avg=False, sync_batchnorm=False, task_name='fb15k-237', terminate_on_nan=False, tpu_cores=None, track_grad_norm=-1, truncated_bptt_steps=None, val_check_interval=1.0, wandb=False, warm_up_radio=0.1, weight_decay=0.01, weights_save_path=None, weights_summary='top')
Global seed set to 7
Load model state dict successful.
14951
unimo.text_embeddings.word_embeddings.weight
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
delete entities without text name.: 100%|█████████████████████████████████████████████████████████████████████████████████████| 272115/272115 [00:00<00:00, 1718344.54it/s]
total entity not in text : 0
max number of filter entities : 4364 954
convert text to examples: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 14951/14951 [00:00<00:00, 272856.79it/s]
0%| | 0/14951 [00:00<?, ?it/s]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/lab401/anaconda3/envs/mkg/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/lab401/anaconda3/envs/mkg/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/home/lab401/MKGformer-main/MKG/data/processor.py", line 554, in encode_lines
enc_lines.append(json.dumps(self.convert_examples_to_features(example=eval(line))))
File "/home/lab401/MKGformer-main/MKG/data/processor.py", line 601, in convert_examples_to_features
assert bpe.mask_token_id in inputs.input_ids, "mask token must in input"
AssertionError: mask token must in input
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "main.py", line 155, in
main()
File "main.py", line 137, in main
trainer.fit(lit_model, datamodule=data)
File "/home/lab401/anaconda3/envs/mkg/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
self._run(model)
File "/home/lab401/anaconda3/envs/mkg/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 713, in _run
self.call_setup_hook(model) # allow user to setup lightning_module in accelerator environment
File "/home/lab401/anaconda3/envs/mkg/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1161, in call_setup_hook
self.datamodule.setup(stage=fn)
File "/home/lab401/anaconda3/envs/mkg/lib/python3.8/site-packages/pytorch_lightning/core/datamodule.py", line 377, in wrapped_fn
return fn(*args, **kwargs)
File "/home/lab401/MKGformer-main/MKG/data/data_module.py", line 228, in setup
self.data_train = get_dataset(self.args, self.processor, self.label_list, self.tokenizer, "train")
File "/home/lab401/MKGformer-main/MKG/data/processor.py", line 68, in wrapper
results = func(*args, **kwargs)
File "/home/lab401/MKGformer-main/MKG/data/processor.py", line 220, in get_dataset
for i, (filt, enc_lines) in tqdm(enumerate(encoded_lines, start=1), total=len(train_examples)):
File "/home/lab401/anaconda3/envs/mkg/lib/python3.8/site-packages/tqdm/std.py", line 1133, in iter
for obj in iterable:
File "/home/lab401/anaconda3/envs/mkg/lib/python3.8/multiprocessing/pool.py", line 420, in
return (item for chunk in result for item in chunk)
File "/home/lab401/anaconda3/envs/mkg/lib/python3.8/multiprocessing/pool.py", line 868, in next
raise value
AssertionError: mask token must in input

您好，

关于M-Encoder中PGI以及CAF的代码储存在哪一个py档呢？

TODO的意思是否需要自己手刻实现？

能否请您开恩提供，下礼拜有一个紧急的会议，求求您了。

Which python version is used to run this repository?

我用仓库提供的默认参数，最高f1只能达到0.86? 论文中报的是0.8749

很抱歉打扰到你，可能我的基础有一些差，为什么训练要分为两阶段呢？

Hi你好，我在跑MRE的部分时代码报了错误，
Traceback (most recent call last):
File "run.py", line 153, in
main()
File "run.py", line 147, in main
trainer.train()
File "/users5/zhzheng/MKGformer-main/MRE/modules/train.py", line 54, in train
(loss, logits), labels = self._step(batch, mode="train")
File "/users5/zhzheng/MKGformer-main/MRE/modules/train.py", line 172, in _step
outputs = self.model(input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids, labels=labels, images=images, aux_imgs=aux_imgs, rcnn_imgs=rcnn_imgs)
File "/users5/zhzheng/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/users5/zhzheng/MKGformer-main/MRE/models/unimo_model.py", line 72, in forward
return_dict=True,)
File "/users5/zhzheng/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/users5/zhzheng/MKGformer-main/MRE/models/modeling_unimo.py", line 721, in forward
return_dict=return_dict,
File "/users5/zhzheng/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/users5/zhzheng/MKGformer-main/MRE/models/modeling_unimo.py", line 620, in forward
current_layer=idx,
File "/users5/zhzheng/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/users5/zhzheng/MKGformer-main/MRE/models/modeling_unimo.py", line 548, in forward
self.feed_forward_chunk, self.chunk_size_feed_forward, self.seq_len_dim, attention_output, fusion_output
File "/users5/zhzheng/anaconda3/envs/py37/lib/python3.7/site-packages/transformers/modeling_utils.py", line 1974, in apply_chunking_to_forward
input_tensor.shape[chunk_dim] == tensor_shape for input_tensor in input_tensors
File "/users5/zhzheng/anaconda3/envs/py37/lib/python3.7/site-packages/transformers/modeling_utils.py", line 1974, in
input_tensor.shape[chunk_dim] == tensor_shape for input_tensor in input_tensors
AttributeError: 'NoneType' object has no attribute 'shape'
我看了一下，应该是由于
modeling_unimo.py的532行附近，

`
self_attention_outputs, fusion_output, qks = self.attention(
            hidden_states,
            attention_mask,
            head_mask,
            output_attentions=output_attentions,
            visual_hidden_state=visual_hidden_state,
            output_qks=output_qks,
            current_layer=current_layer,
        )
        attention_output = self_attention_outputs[0]
        outputs = self_attention_outputs[1:]  # add self attentions if we output attention weights
`

其中fusion output为None，这个是由于visual_hidden_state为None导致的。 但是由于UniEncoder的代码逻辑，

# text
            # TODO: 9-12 layers past vison qks to text
            last_hidden_state = vision_hidden_states if idx >= 8 else None
            output_qks = True if idx >= 7 else None
            layer_head_mask = head_mask[idx] if head_mask is not None else None
            text_layer_module = self.text_layer[idx]
            text_layer_output = text_layer_module(
                    text_hidden_states,
                    attention_mask=attention_mask,
                    head_mask=layer_head_mask,
                    visual_hidden_state=last_hidden_state,
                    output_attentions=output_attentions,
                    output_qks=output_qks,
                    current_layer=idx,
            )

，第9-12层传入的visual_hidden_state为空，这就导致了报错，能帮我看一下这个怎么解决吗，非常感谢~

Hi, I recently want to complete a MMKG in my own datasets. Specifically, I want to use your MKG code. However, I can't donwnload the image data for FB-15K-237 and WN18, I also can't run the main function in the MKG directory.

Is there more information about how to download image data and where to store it? If we run your multi-modal knowledge graph, that will make it easier for me to reconstruct my own multi-modal entity-relationship dataset.

Thanks!!

您好，我从提供的Google Drive中下载了wn18-images，发现只有7,801个实体有图片文件夹，不知道是不是没有上传完整呢？

您好，我下载了百度网盘上的FB15K数据集，运行bash scripts/pretrain_fb15k-237-image.sh后，在Validation sanity check阶段报了ValueError: mean must have 1 elements if it is an iterable, got 3 错误，想请教一下该如何解决。具体报错信息如下：

Validation sanity check: 50%|█████████████████████████████████████████████████████▌ | 1/2 [00:03<00:03, 3.37s/it]
Traceback (most recent call last):
File "main.py", line 153, in
main()
File "main.py", line 135, in main
trainer.fit(lit_model, datamodule=data)
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
self._run(model)
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 756, in _run
self.dispatch()
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 797, in dispatch
self.accelerator.start_training(self)
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
self.training_type_plugin.start_training(trainer)
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
self._results = trainer.run_stage()
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 807, in run_stage
return self.run_train()
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 842, in run_train
self.run_sanity_check(self.lightning_module)
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1107, in run_sanity_check
self.run_evaluation()
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 949, in run_evaluation
for batch_idx, batch in enumerate(dataloader):
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 630, in next
data = self._next_data()
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1325, in _next_data
return self._process_data(data)
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1373, in _process_data
data.reraise()
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/torch/_utils.py", line 694, in reraise
raise exception
ValueError: Caught ValueError in DataLoader worker process 1.
Original Traceback (most recent call last):
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
return self.collate_fn(data)
File "/home/user1/chenchi/MKGformer/MKG/data/data_module.py", line 168, in call
aux_img = aux_processor(images=aux_img, return_tensors='pt')['pixel_values'].squeeze()
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/transformers/models/clip/processing_clip.py", line 104, in call
image_features = self.image_processor(images, return_tensors=return_tensors, **kwargs)
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/transformers/image_processing_utils.py", line 552, in call
return self.preprocess(images, **kwargs)
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/transformers/models/clip/image_processing_clip.py", line 316, in preprocess
images = [
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/transformers/models/clip/image_processing_clip.py", line 317, in
self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/transformers/image_processing_utils.py", line 622, in normalize
return normalize(
File "/home/user1/anaconda3/envs/MKG/lib/python3.8/site-packages/transformers/image_transforms.py", line 386, in normalize
raise ValueError(f"mean must have {num_channels} elements if it is an iterable, got {len(mean)}")
ValueError: mean must have 1 elements if it is an iterable, got 3

您好，请问TransformerLitModel类中的training_step方法，labels和label分别表示什么意思呢？两者名称太相似，有点难理解。

Thanks for your graceful code! I noticed that u have released network disk links for multimodal KG data，but I have no access to the password，could u tell me that? Thanks~

可以提供一下“openai/clip-vit-base-patch32"的下载链接吗？
我已经联网了，但不知道为何服务器下载不下来？

我在加载MKG预训练模型时反复报如下错误：
pytorch_lightning.utilities.exceptions.MisconfigurationException: CUDAAccelerator can not run on your system since the accelerator is not available. The following accelerator(s) is available and can be passed into accelerator argument of Trainer: ['cpu'].
但是我的本机是有CUDA和GPU的，请问有没有遇到同样问题的朋友？求解答！
谢谢！

作者，您好，最近做实验的时候用了gitHub给的网盘中提供的IMG数据集，但是发现实体数目对应不上，以WN18RR为例，实体的数目应该有40943个，但是WN18-IMG对应的文件夹只有7801个，请问是不是有的实体没有对应的图片，然后处理的时候需要过滤掉没有图片的实体呢，这样做有没有什么影响，期待您的回复，谢谢

你好，在MKG任务中，main.py中的参数model_class设置为RobertaUseLabelWord，但是并没有在models文件中找到对应的类

我发现你们的图片数据集与RSME好像不太一致，以FB15k为例，在你们说给出的百度硬盘提供的数据集只有bing的，但是readme中给出的另一种方法 mmkb 中所下载的数据集同时包括了必应谷歌和雅虎下载的数据集

我也细致地比较了RSME和你们代码中datasets，发现确实有点不一样，我知道可能由于国内上谷歌和雅虎不太方便，但还是想搞清楚，你们的图片数据集（百度网盘提供的）和RSME是否不一致，或者说与你们在readme提供的第一种需要自己用脚本下载的图片数据集不一致

如果是不一致的，那么在训练的时候，你们实际使用的是提供的百度网盘中的数据集，还是第一种需要自己用脚本下载的数据集呢