Comments (11)
Roshanson
commented on August 11, 2024
1
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
在 kiwi/metrics/functions.py:96 加上如下代码
if not isinstance(hat_y, list):
hat_y = hat_y.cpu()
if not isinstance(y, list):
y = y.cpu()
PicklingError: Can't pickle <class 'kiwi.data.encoders.wmt_qe_data_encoder.InputFields[PositiveInt]'>: attribute lookup InputFields[PositiveInt] on kiwi.data.encoders.wmt_qe_data_encoder failed
这个问题在使用多块GPU的时候我也遇到了。。 也不知道怎么改 = =
from openkiwi.
zhaofuchen
commented on August 11, 2024
对齐文件是必须的，否则无法训练。你可以使用下面项目制作对齐文件：
https://github.com/Unbabel/word-level-qe-corpus-builder
from openkiwi.
leileilin
commented on August 11, 2024
font{
line-height: 1.6;
}
ul,ol{
padding-left: 20px;
list-style-position: inside;
}
你好，我已经在有对齐文件的条件下训练了nuqe模型，但是为什么预测的时候也要对齐吗？
因为我在预测源句子和翻译句子的时候报了一样的错误。非常感谢！
1299192319
***@***.***
签名由
网易邮箱大师
定制
On 3/17/2021 ***@***.***> wrote：
对齐文件是必须的，否则无法训练。你可以使用下面项目制作对齐文件：
https://github.com/Unbabel/word-level-qe-corpus-builder
—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.
from openkiwi.
zhaofuchen
commented on August 11, 2024
训练和预测输入的数据是一样的，本质上预测就是使用已训练好的模型参数。
所以预测也需要对齐信息。
from openkiwi.
leileilin
commented on August 11, 2024
font{
line-height: 1.6;
}
ul,ol{
padding-left: 20px;
list-style-position: inside;
}
那我问一下，就是OpenKiwi框架里的所有模型的训练和预测输入的数据都需要对齐数据吗？还是只有nuqe需要？非常感谢！
1299192319
***@***.***
签名由
网易邮箱大师
定制
On 3/17/2021 ***@***.***> wrote：
训练和预测输入的数据是一样的，本质上预测就是使用已训练好的模型参数。
所以预测也需要对齐信息。
—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.
from openkiwi.
zhaofuchen
commented on August 11, 2024
全部都需要，个人建议你别折腾nuqe了，这个模型效果没有其他的好。还有就是 estimator-predictor模型预训练官方有训练好的模型，不要自己训练。你找找看，我不记得了。
BERT，XLM，XLM-Roberta这三个模型是基于transformers实现的，目前效果比较好。
from openkiwi.
leileilin
commented on August 11, 2024
好的，谢谢！------------------ 原始邮件 ------------------
***@***.***>
发送时间: 2021年3月17日(星期三) 下午2:28
***@***.***>;
***@***.******@***.***>;
主题: Re: [Unbabel/OpenKiwi] some problems about data without alignments (#95)
from openkiwi.
leileilin
commented on August 11, 2024
font{
line-height: 1.6;
}
ul,ol{
padding-left: 20px;
list-style-position: inside;
}
你好，当我用BERT和XLM-R训练的时候，总是会报错：
PicklingError: Can't pickle <class 'kiwi.data.encoders.wmt_qe_data_encoder.InputFields[PositiveInt]'>: attribute lookup InputFields[PositiveInt] on kiwi.data.encoders.wmt_qe_data_encoder failed你知道这是因为什么吗？非常感谢！
1299192319
***@***.***
签名由
网易邮箱大师
定制
On 3/17/2021 ***@***.***> wrote：
全部都需要，个人建议你别折腾nuqe了，这个模型效果没有其他的好。还有就是 estimator-predictor模型预训练官方有训练好的模型，不要自己训练。你找找看，我不记得了。
BERT，XLM，XLM-Roberta这三个模型是基于transformers实现的，目前效果比较好。
—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.
from openkiwi.
leileilin
commented on August 11, 2024
font{
line-height: 1.6;
}
ul,ol{
padding-left: 20px;
list-style-position: inside;
}
不好意思，已经解决了，当我用gpu训练的时候不会报这个错误。
1299192319
***@***.***
签名由
网易邮箱大师
定制
On 3/22/2021 ***@***.***> wrote：
font{
line-height: 1.6;
}
ul,ol{
padding-left: 20px;
list-style-position: inside;
}
你好，当我用BERT和XLM-R训练的时候，总是会报错：
PicklingError: Can't pickle <class 'kiwi.data.encoders.wmt_qe_data_encoder.InputFields[PositiveInt]'>: attribute lookup InputFields[PositiveInt] on kiwi.data.encoders.wmt_qe_data_encoder failed你知道这是因为什么吗？非常感谢！
1299192319
***@***.***
签名由
网易邮箱大师
定制
On 3/17/2021 ***@***.***> wrote：
全部都需要，个人建议你别折腾nuqe了，这个模型效果没有其他的好。还有就是 estimator-predictor模型预训练官方有训练好的模型，不要自己训练。你找找看，我不记得了。
BERT，XLM，XLM-Roberta这三个模型是基于transformers实现的，目前效果比较好。
—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.
from openkiwi.
leileilin
commented on August 11, 2024
font{
line-height: 1.6;
}
ul,ol{
padding-left: 20px;
list-style-position: inside;
}
font{
line-height: 1.6;
}
ul,ol{
padding-left: 20px;
list-style-position: inside;
}
不好意思，已经解决了，当我用cpu训练时不会报这个错误，但是当我用一个gpu训练时会报错：TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
1299192319
***@***.***
签名由
网易邮箱大师
定制
On 3/22/2021 ***@***.***> wrote：
font{
line-height: 1.6;
}
ul,ol{
padding-left: 20px;
list-style-position: inside;
}
你好，当我用BERT和XLM-R训练的时候，总是会报错：
PicklingError: Can't pickle <class 'kiwi.data.encoders.wmt_qe_data_encoder.InputFields[PositiveInt]'>: attribute lookup InputFields[PositiveInt] on kiwi.data.encoders.wmt_qe_data_encoder failed你知道这是因为什么吗？非常感谢！
1299192319
***@***.***
签名由
网易邮箱大师
定制
On 3/17/2021 ***@***.***> wrote：
全部都需要，个人建议你别折腾nuqe了，这个模型效果没有其他的好。还有就是 estimator-predictor模型预训练官方有训练好的模型，不要自己训练。你找找看，我不记得了。
BERT，XLM，XLM-Roberta这三个模型是基于transformers实现的，目前效果比较好。
—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.
from openkiwi.
zhaofuchen
commented on August 11, 2024
抱歉额，我也没遇到过，帮不了你。
…------------------ 原始邮件 ------------------
发件人: "Unbabel/OpenKiwi" ***@***.***>;
发送时间: 2021年4月21日(星期三) 下午5:47
***@***.***>;
***@***.******@***.***>;
主题: Re: [Unbabel/OpenKiwi] some problems about data without alignments (#95)
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
在 kiwi/metrics/functions.py:96 加上如下代码
if not isinstance(hat_y, list): hat_y = hat_y.cpu() if not isinstance(y, list): y = y.cpu()
PicklingError: Can't pickle <class 'kiwi.data.encoders.wmt_qe_data_encoder.InputFields[PositiveInt]'>: attribute lookup InputFields[PositiveInt] on kiwi.data.encoders.wmt_qe_data_encoder failed
这个问题在使用多块GPU的时候我也遇到了。。 也不知道怎么改 = =
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
from openkiwi.
Related Issues (20)
- TypeError: cannot unpack non-iterable NoneType object HOT 1
- The prediction process is not complete by Predictor Estimator. HOT 5
- OpenKiwi always download the tokenizer files for XLMRoberta even if a local path is configured. HOT 2
- Do openKiwi have confident score? HOT 1
- Error Pre-Training Predictor: "model -> encoder -> encode_source extra fields not permitted (type=value_error.extra)" HOT 1
- some confusions
- pkgutil.iter_modules() error: 'PosixPath' object has no attribute 'startswith'
- Got exception when import kiwi
- Seems that maximum token support for a sentence is 512?
- PicklingError: Can't pickle <class 'kiwi.data.encoders.wmt_qe_data_encoder.InputFields[PositiveInt]'>: attribute lookup InputFields[PositiveInt] on kiwi.data.encoders.wmt_qe_data_encoder failed HOT 2
- Do you need to tokenize your data when using a BERT/ROBERTA model?
- Pretrain config file
- What are source_pos and target_pos in the train_config.yaml?
- Why does it need "--model" paramter when I give a specific config? HOT 2
- What languages do the OpenKiwi support?
- I suppose that the code comment should be remove. HOT 2
- Error at Predictor Training: "Predictor is not a subclass of QESystem" HOT 2
- OSError: Can't load weights for 'xlm-roberta-base'. HOT 16
- open cannot unpack non-iterable NoneType object HOT 16
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openkiwi.