fastnlp / tener Goto Github PK

使用中文数据集，train.txt大小50M，BIO标注法，更改数据路径和模型路径，以及BSEM为BIO,其他没有变动。使用python train_tenser_cn.py dataset ontonotes 即使是batchsize=1，仍会报错。
RuntimeError: CUDA out of memory. Tried to allocate 57.83 GiB (GPU 0; 8.00 GiB total capacity
windows和Linux测试，出现同样问题。
torch=1.5.1
fastnlp=0.5.5
prettytable=0.7.2

作者你好！

尝试在微博数据上复现结果时出现使用DataSetGetter的__getattr__时出现无限递归问题，尝试修改为字典方法也无效，不知道程序是否在windows上测试过？用的fastNLP版本为5.5，尝试了1.1和1.5.1版本的torch都没有解决这个问题。

D:\Anaconda\envs\torch_env\python.exe E:/BoXiao/TENER-master/train_tener_cn.py --dataset weibo
Read cache from caches/weibo_transformer_bmeso_True.pkl.
In total 3 datasets:
train has 1350 instances.
dev has 270 instances.
test has 270 instances.
In total 3 vocabs:
chars has 3356 entries.
bigrams has 42184 entries.
target has 29 entries.

input fields after batch(if batch size is 2):
target: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2, 26])
chars: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2, 26])
bigrams: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2, 26])
seq_len: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2])
target fields after batch(if batch size is 2):
target: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2, 26])
seq_len: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2])

training epochs started 2020-12-07-18-47-38-732204
Epoch 1/100: 0%| | 0/8500 [00:00<?, ?it/s, loss:{0:<6.5f}]Read cache from caches/weibo_transformer_bmeso_True.pkl.
In total 3 datasets:
train has 1350 instances.
dev has 270 instances.
test has 270 instances.
In total 3 vocabs:
chars has 3356 entries.
bigrams has 42184 entries.
target has 29 entries.

Traceback (most recent call last):
File "", line 1, in
File "D:\Anaconda\envs\torch_env\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "D:\Anaconda\envs\torch_env\lib\multiprocessing\spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "D:\Anaconda\envs\torch_env\lib\site-packages\fastNLP\core\batch.py", line 100, in getattr
if hasattr(self.dataset, item):
File "D:\Anaconda\envs\torch_env\lib\site-packages\fastNLP\core\batch.py", line 100, in getattr
if hasattr(self.dataset, item):
File "D:\Anaconda\envs\torch_env\lib\site-packages\fastNLP\core\batch.py", line 100, in getattr
if hasattr(self.dataset, item):
[Previous line repeated 662 more times]
RecursionError: maximum recursion depth exceeded
Read cache from caches/weibo_transformer_bmeso_True.pkl.
In total 3 datasets:
train has 1350 instances.
dev has 270 instances.
test has 270 instances.
In total 3 vocabs:
chars has 3356 entries.
bigrams has 42184 entries.
target has 29 entries.

Traceback (most recent call last):
File "E:/BoXiao/TENER-master/train_tener_cn.py", line 130, in
trainer.train(load_best_model=False)
File "D:\Anaconda\envs\torch_env\lib\site-packages\fastNLP\core\trainer.py", line 618, in train
raise e
File "D:\Anaconda\envs\torch_env\lib\site-packages\fastNLP\core\trainer.py", line 611, in train
self._train()
File "D:\Anaconda\envs\torch_env\lib\site-packages\fastNLP\core\trainer.py", line 658, in _train
for batch_x, batch_y in self.data_iterator:
File "D:\Anaconda\envs\torch_env\lib\site-packages\fastNLP\core\batch.py", line 267, in iter
for indices, batch_x, batch_y in self.dataiter:
File "D:\Anaconda\envs\torch_env\lib\site-packages\torch\utils\data\dataloader.py", line 193, in iter
return _DataLoaderIter(self)
File "D:\Anaconda\envs\torch_env\lib\site-packages\torch\utils\data\dataloader.py", line 493, in init
Traceback (most recent call last):
File "", line 1, in
self._put_indices()
File "D:\Anaconda\envs\torch_env\lib\site-packages\torch\utils\data\dataloader.py", line 591, in _put_indices
File "D:\Anaconda\envs\torch_env\lib\multiprocessing\spawn.py", line 105, in spawn_main
indices = next(self.sample_iter, None)
File "D:\Anaconda\envs\torch_env\lib\site-packages\torch\utils\data\sampler.py", line 172, in iter
exitcode = _main(fd)
File "D:\Anaconda\envs\torch_env\lib\multiprocessing\spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "D:\Anaconda\envs\torch_env\lib\site-packages\fastNLP\core\batch.py", line 100, in getattr
if hasattr(self.dataset, item):
File "D:\Anaconda\envs\torch_env\lib\site-packages\fastNLP\core\batch.py", line 100, in getattr
if hasattr(self.dataset, item):
File "D:\Anaconda\envs\torch_env\lib\site-packages\fastNLP\core\batch.py", line 100, in getattr
if hasattr(self.dataset, item):
[Previous line repeated 662 more times]
RecursionError: maximum recursion depth exceeded
for idx in self.sampler:
File "D:\Anaconda\envs\torch_env\lib\site-packages\fastNLP\core\batch.py", line 121, in iter
return iter(self.sampler(self.dataset))
File "D:\Anaconda\envs\torch_env\lib\site-packages\fastNLP\core\sampler.py", line 79, in call
seq_lens = data_set.get_all_fields()[self.seq_len_field_name].content
File "D:\Anaconda\envs\torch_env\lib\site-packages\fastNLP\core\batch.py", line 101, in getattr
return object.getattr(self.dataset, item)
AttributeError: type object 'object' has no attribute 'getattr'

期待回复，万分感谢。

您好，请问fastnlp中下载的peopleDaily是哪里的数据呢

请问代码relative_transformer.py种127行的E_如何理解，看了论文中在计算attention score时似乎没有这一项？

作者你好，请问OntoNotes4.0是在哪里下载呢，我只看到https://github.com/yhcc/OntoNotes-5.0-NER
RuntimeError: Ontonotes cannot be downloaded automatically, you can refer https://github.com/yhcc/OntoNotes-5.0-NER to download and preprocess.

你好，请问在程序中为什么训练集是train_dev，验证集和测试集都是test呢

Trainer的sava_path不为None的时候，会出现：
UserWarning: Couldn't retrieve source code for container of type ConditionalRandomField. It won't be checked for correctness upon loading.
虽然也保存了模型，但是加载之后会报错，请问这个怎么处理呢？

请教relative_transformer.py中_transpose_shift函数中的几个问题：
（1）它实现了矩阵的什么变换（如移动、旋转等）？
（2）怎么理解它是如何实现这种变换的呢？
（3）倒数第3行中的indice为什么只选取奇数行呢？
（4）_transpose_shift函数与_shift有什么区别，又有什么联系？
十分感谢

def _transpose_shift(self, E): 
       #E=[B,N,L,2*L]=[bsz, head, max_len, 2max_len] 如[2, 4, 68, 136]；
        bsz, n_head, max_len, _ = E.size()
        zero_pad = E.new_zeros(bsz, n_head, max_len, 1)
        E = torch.cat([E, zero_pad], dim=-1).view(bsz, n_head, -1, max_len) # [B,N,2L,L] 
        indice = (torch.arange(max_len) * 2 + 1).to(E.device) # 选取是奇数行：[1,3,5...135]
        E = E.index_select(index=indice, dim=-2).transpose(-1, -2)  
        return E

hello，How can the TENER solve the OOV problem, please?

As above, can TENER preprocessing be done on dataset that does not follow CONLL-2003 format? My dataset does not have BIO scheme tagging. Meaning the sentences will look like this.

sentence = ['Hi', 'I', 'study', 'in', 'China', 'and', 'work' , 'in', 'ABC']
tag = ['O', 'O', 'O', 'O', 'Country', 'O', 'O', 'O', 'Company']

i want to load model TENER after trained for predict
predict with input type Str
please!

Thanks very much for sharing the code! When I tried to reproduce the msra result, I found the msra dataset only contained train and test set. So, how do your split the train set into train and dev set? It seems the paper does not mention it. I will be grateful if you could reply.

您好，我的数据集已经处理为了如下形式：
口 O
腔 O
溃 O
疡 O
加 O
上 O
如何进一步将数据集处理为‘train.char.bmes'形式，让模型run起来呢？

from fastNLP.io.loader import WeiboNERLoader
save_path = WeiboNERLoader().download()
print(save_path) # 查看文件在本地的存放路径

作者您好，有关weibo数据集，请问您在论文中使用的数据集是weiboNER.conll.train还是weiboNER_2nd_conll.train呢？

很棒的工作，解决了我之前Transformer在NER任务上不work的问题
本人复现了TF版本的代码，如果代码存在问题请指出（简单在中文NER任务上测试，取得与BiLSTM相近的性能，没有严格调参）
https://github.com/qq547276542/TF_adapting_transformer

modules文件夹下transformer.py中用于实现qkv矩阵的线性层，bias参数应该是False吧

class MultiHeadAttn(nn.Module):
    def __init__(self, d_model, n_head, dropout=0.1, scale=False):
        super().__init__()
        assert d_model%n_head==0

        self.n_head = n_head
        self.qkv_linear = nn.Linear(d_model, 3*d_model)
        self.fc = nn.Linear(d_model, d_model)

在relative_transformer.py文件中qv的实现，bias参数是False的

class RelativeMultiHeadAttn(nn.Module):
    def __init__(self, d_model, n_head, dropout, r_w_bias=None, r_r_bias=None, scale=False):
        super().__init__()
        self.qv_linear = nn.Linear(d_model, d_model * 2, bias=False)
        self.n_head = n_head
        self.head_dim = d_model // n_head
        self.dropout_layer = nn.Dropout(dropout)

Hi, have you compared adapted transformer with bert, where pre-trained knowledge might make up for the drawback of vanilla transformer?

作者你好，这两天在阅读代码，看到将character和tag转换为index这里，我在想如果我想得到预测的BIO标签，那我怎么把index映射回去呢。

您好，我正在学习您的tener。里面发现WeiboNER/train.all.bmes 这种格式的文件。缺少。可是我找了一下，weibo的 标注，没看的这种标注的。
介绍里的 是
科 O
技 O
全 O
方 O
位 O
资 O
讯 O
智 O

可是 bmes是什么呢？怎么获取，或怎么标注出来？
谢谢

@misc{yan2019tener,
    title={TENER: Adapting Transformer Encoder for Named Entity Recognition},
    author={Hang Yan and Bocao Deng and Xiaonan Li and Xipeng Qiu},
    year={2019},
    eprint={1911.04474},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

Is it right?

Hi,

Could you please give clarity on the significance of using 'num_embeddings' and 'origin_shift' when computing RelativeSinusoidalPositionalEmbedding?

self.origin_shift = num_embeddings//2 + 1

Thanks,
Neena

if tag_set == bio_tag_set or tag_set==set('o'):
    return 'bio'

原来标签不能只有“O”

Hi,

Thanks for your amazing work.

For the following unigram and bigram embedding, If I wanna try my own set of vec files, what can i do? T

For the Chinese datasets, you can download the pretrained unigram and bigram embeddings in Baidu Cloud. Download the 'gigaword_chn.all.a2b.uni.iter50.vec' and 'gigaword_chn.all.a2b.bi.iter50.vec'. Then replace the embedding path in train_tener_cn.py

Thanks a lot,

Marcus

sorry I am not able to prepare the dateset since it is mainly explained is Chinese. can you please help to prepare or download the prepared version?

如题

感谢您的工作！我在运行时想请教一个问题，论文中的实验结果是选取的'may not correspond to the best dev performance' 还是‘correspond to the best dev performance’。谢谢！！

我想问下 对于测试集长度长于训练集的训练长度的文本时？对于这部分文本长度比训练集设定的最大训练文本长度长的话，在计算P\R\F1时是怎么进行处理的呀？
是截断成两个句子分别输入模型预测吗？
还是对于测试集的文本 长于训练集长度的部分进行抛弃，计算PRF1时候也不用它们？
比如 训练集 最大长度为128，测试集长度是200，那么在计算P R F1时是怎么计算的呢?
是将测试集截断成128从而与预测的128长度进行计算吗

Thanks for sharing the code,
can you please share arguments that has been used for training, specifically in the paper propose a modified version of positional encoding but I see that it is set to use None in the file.
Thanks for your help.

assert tag in tags, f"{tag} is not a valid tag in encoding type:{encoding_type}. Please check your "
AssertionError: i is not a valid tag in encoding type:bmeso. Please check your encoding_type.
是哪里有问题呢？

Hello, first and foremost I hope this message finds you well and healthy. I need to load the model and test it on specific data. The data type is str. Can you please send me the demo? Thank you.
I look forward to your reply.
Sincerely,

作者大大您好，我想问一下运行过程出现两个结果，一个是test-data后的，一个是dev后的。论文中的结果是取哪一个在6个数据及上和其他模型对比呢？还有一般不都是取test吗？为什么最后是get best dev？期待您的回答！

Linux和Windows下，使用CPU训练时，可设置num_workers=0

trainer = Trainer(data_bundle.get_dataset('train'), model, optimizer, batch_size=batch_size, sampler=BucketSampler(), num_workers=0, n_epochs=n_epochs, dev_data=data_bundle.get_dataset('dev'), metrics=SpanFPreRecMetric(tag_vocab=data_bundle.get_vocab('target'), encoding_type=encoding_type), dev_batch_size=batch_size, callbacks=callbacks, device=device, test_use_tqdm=False, use_tqdm=True, print_every=300, save_path=None)

将代码放到Linux平台，使用自己的BIO数据，报“UnicodeDecoderError”，但是我检查了数据集，字和标签之间用空格和\t都测试了，还是报相同的错误，请问有什么办法可以解决吗。

你好，我用自己的数据集训练，loss下降很少，100epoch训练的评分SpanFPreRecMetric: f=0.0, pre=0.0, rec=0.0 ，loss为113.这是什么问题？是数据集的问题吗？数据集格式为BIO格式