smilelight / lightnlp Goto Github PK

View Code? Open in Web Editor NEW

824.0 31.0 213.0 50.49 MB

基于Pytorch和torchtext的自然语言处理深度学习框架。

License: Apache License 2.0

Python 100.00%

nlp pytorch torchtext srl deep-learning cws pos relation-extraction dependency-parsing sentence-similarity

lightnlp's Introduction

Hi there 👋

😄 Interested in natural language processing and knowledge graph

lightnlp's People

Contributors

Stargazers

Watchers

Forkers

zhuzhuxa coolcoolhua fendaq helen0804 skywindy hughimr colinsongf thinkerboy middle-plat-ai 90217 xucong053 boluoyu wendonggan silencewinter homelh gandalfliu zhongwen7710 w6688j cslele tomlee20180103 gttttttt babygirlgtt charlottesean aimerzh mojirml wxrui zhiyuxzh miaotxy henryzhou1002 nwf5d frankvlee thoye delaiahz hatleon githubbayes hunterkai lxz1992 buaachuanwang stonetingxin wuyunxiangwyx arsentiii likeucode allensmile legendtianjin chagge benjamesbabala liu-zhenhui lichao88 sunjing123 blacog juvu cnlpt melonxi lhzing see-u-see davidhdw pincute pengyuange yw1991 letanqmin xiangbei1997 wangxinyufighting christinexinxiang shenxinli askintution liuweiping2020 scievan chenliy 2448795365 dongyifeng shijie97 yuxuan2015 475189541 ccq666666 xiejinglover lzgh liboge zixuedanxin zwjwhxz tianfrank hjqjet mzss0 lavine24 hanktown newverygood xiaodao010 liangliang12 3sunny gm19900510 jangocheng gurusmile liuyibo2017 xiaoying1 ddzhen eltheren jxh4945777 sonehh yanghaihuo youarerare choufeng2018

lightnlp's Issues

token_vec_300.bin is a w2v or fasttext format?

Thanks for your amazing Library

One simple question about your NER module

vec_path = '/home/lightsmile/NLP/embedding/char/token_vec_300.bin'

The vec bin is a w2v or fasttext file?

I already had my w2v pre-trained bin file because its size is only about 200M.

If fasttext bin file, it is always about 1G , not too capable for production environment.

Thanks a lot.

Traceback (most recent call last):
File "", line 1, in
File "/home/bb0/anaconda3/lib/python3.6/site-packages/lightnlp/sr/init.py", line 2, in
from .ss.module import SS
File "/home/bb0/anaconda3/lib/python3.6/site-packages/lightnlp/sr/ss/module.py", line 2, in
import torch
File "/home/bb0/anaconda3/lib/python3.6/site-packages/torch/init.py", line 84, in
from torch._C import *
ImportError: libcurand.so.8.0: cannot open shared object file: No such file or directory

我安装的是cuda9.0

emissions and tags not match

the first two dimensions of emissions and tags must match, got (212, 128) and (204, 128)

吐槽一下

吐槽一下：这个line如果是以空格开头的，sc-ner模型都跑不起来。看来还是得找正规点的模型，坑有点多，本来提供一个数据就可以直接跑了，结果调试模型代码半天，太浪费时间了。。。

你的代码就是垃圾！！数据格式一样都报错

垃圾以后别装逼了祝你考试失败垃圾

你们分词跑多少行数据

我怎么数据5万行比一万行准确率还低

测试的错误

您好：
使用你的代码测试了一下出了一下错误：ModuleNotFoundError: No module named 'lightnlp.sl.pos.utils'，请问这个错误是怎么导致的？
谢谢

语义相似度模型用第一层的隐藏状态计算相似度？是不是有问题？

sr/ss/model.py

    def forward(self, left, right):
        left_vec = self.embedding(left.to(DEVICE)).to(DEVICE)
        #         left_vec = pack_padded_sequence(left_vec, left_sent_lengths)
        right_vec = self.embedding(right.to(DEVICE)).to(DEVICE)
        #         right_vec = pack_padded_sequence(right_vec, right_sent_lengths)

        self.hidden = self.init_hidden(batch_size=left.size(1))

        left_lstm_out, (left_lstm_hidden, _) = self.lstm(left_vec, self.hidden)
        #         left_lstm_out, left_batch_size = pad_packed_sequence(left_lstm_out)
        #         assert torch.equal(left_sent_lengths, left_batch_size.to(DEVICE))

        right_lstm_out, (right_lstm_hidden, _) = self.lstm(right_vec, self.hidden)
        #         right_lstm_out, right_batch_size = pad_packed_sequence(right_lstm_out)
        #         assert torch.equal(right_sent_lengths, right_batch_size.to(DEVICE))

        return self.manhattan_distance(left_lstm_hidden[0], right_lstm_hidden[0])

最后一行计算曼哈顿距离 self.manhattan_distance(left_lstm_hidden[0], right_lstm_hidden[0])，用的是第一层的隐藏状态，是不是有问题？

文本生成，语言模型示例报错 ModuleNotFoundError: No module named 'lightnlp.tg.cb.models'

Mac 最新版本，按照Readme完成安装。
example目录执行python3 ./test_lm.py，报错：

Traceback (most recent call last):
  File "./test_lm.py", line 5, in <module>
    from lightnlp.tg import LM
  File "/usr/local/lib/python3.7/site-packages/lightnlp/tg/__init__.py", line 3, in <module>
    from .cb.module import CB
  File "/usr/local/lib/python3.7/site-packages/lightnlp/tg/cb/module.py", line 16, in <module>
    from .model import CBConfig, CBSeq2Seq
  File "/usr/local/lib/python3.7/site-packages/lightnlp/tg/cb/model.py", line 2, in <module>
    from .models.encoder import Encoder
ModuleNotFoundError: No module named 'lightnlp.tg.cb.models'

数据集

可以麻烦你提供一下数据集吗？或者数据集的链接

商品名称分类

我想做一个按商品名称对商品进行分类的功能，请问用你这个训练模型合适吗

您好如何在摘要生成时使用自定义的训练集呢

我是一个新手参照您的dev_path = '../data/ts/test.sample.tsv'下的格式生成了训练集但是会报错属性缺失如下：
File "/Users/hanyang/Downloads/lightNLP-master/examples/test_ts.py", line 20, in
log_dir='/Users/hanyang/Desktop')
File "/Users/hanyang/Downloads/lightNLP-master/lightnlp/tg/ts/module.py", line 31, in train
word_vocab = ts_tool.get_vocab(train_dataset, dev_dataset)
File "/Users/hanyang/Downloads/lightNLP-master/lightnlp/tg/ts/tool.py", line 41, in get_vocab
TEXT.build_vocab(*dataset)
File "/Users/hanyang/Library/Python/3.7/lib/python/site-packages/torchtext/data/field.py", line 298, in build_vocab
for x in data:
File "/Users/hanyang/Library/Python/3.7/lib/python/site-packages/torchtext/data/dataset.py", line 154, in getattr
yield getattr(x, attr)
AttributeError: 'Example' object has no attribute 'summarization'

[Errno 2] No such file or directory: './lm_saves/config.pkl'

你好，[Errno 2] No such file or directory: './lm_saves/config.pkl' 这里config.pkl文件是什么