ymcui / pert Goto Github PK
View Code? Open in Web Editor NEWPERT: Pre-training BERT with Permuted Language Model
Home Page: https://arxiv.org/abs/2203.06906
License: Apache License 2.0
PERT: Pre-training BERT with Permuted Language Model
Home Page: https://arxiv.org/abs/2203.06906
License: Apache License 2.0
想问下,如何在自己语料上进行预训练,是按照之前mask那种方式直接预训练吗。数据处理和模型源码会开放吗
该模型可以实现两个句子的次序预测吗
请问下这个代码也是不开源的吧
你好,可以开源一下文本纠错(乱序)有关数据集和源码吗,想学习一下,谢谢
你好,当我用hfl/chinese-pert-base-mrc或large进行阅读理解时,尚未进行微调,文本为“我叫沃尔夫冈,我住在柏林。”,问题为“我弟弟住在哪里”,此时得到的答案为“柏林”。
当我把问题改为“我在哪里工作”、“我弟弟住在哪里”、“我来自哪里”、“我妹妹来自哪里”等,得到答案也是柏林,概率也都是0.9多,请问这种情况该如何改善呢?
Hi,
Is the shape of p_i, b in equation 5 should be in R^{k} not R^{L}?
In the paper https://arxiv.org/pdf/2203.06906.pdf
Cui老师您好!
看到的论文您的 [2203.06906] PERT: Pre-training BERT with Permuted Language Model,对我很有帮助,并正在复现它。过程中,有一个细节不明白,希望您能不吝指点。
对于公式(5),L是最大长度N吗?如下图。前面您提到L是transfomer的层数,不能理解为什么层数L和p_i有关。
H^{~}_{i}的维度是(1,768), H^{T}维度是 (768,N), 那么 b 的维度应该是(N)。
希望能尽快得到您的回复,谢谢。
close
在对模型进一步针对数据集训练时,给出了警告:
UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
return fn(*args, **kwargs)
UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:555.)
attn_output = torch.nn.functional.scaled_dot_product_attention(
FutureWarning: torch.cpu.amp.autocast(args...)
is deprecated. Please use torch.amp.autocast('cpu', args...)
instead.
with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs): # type: ignore[attr-defined]
是否可以给一些指导,谢谢!
请问pert-large预训练时学习率是多少?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.