Comments (5)
@donglixp , can you please provide details regarding how to run these 3 steps?
def process_detokenize(chunk):
twd = TreebankWordDetokenizer()
tokenizer = BertTokenizer.from_pretrained(
args.bert_model, do_lower_case=args.do_lower_case)
r_list = []
for idx, line in chunk:
line = line.strip().replace('``', '"').replace('\'\'', '"').replace('`','\'')
s_list = [twd.detokenize(x.strip().split(
' '), convert_parentheses=True) for x in line.split('<S_SEP>')]
tk_list = [tokenizer.tokenize(s) for s in s_list]
r_list.append((idx, s_list, tk_list))
return r_list
def read_tokenized_file(fn):
with open(fn, 'r', encoding='utf-8') as f_in:
l_list = [l for l in f_in]
num_pool = min(args.processes, len(l_list))
p = Pool(num_pool)
chunk_list = partition_all(
int(len(l_list)/num_pool), list(enumerate(l_list)))
r_list = []
with tqdm(total=len(l_list)) as pbar:
for r in p.imap_unordered(process_detokenize, chunk_list):
r_list.extend(r)
pbar.update(len(r))
p.close()
p.join()
r_list.sort(key=lambda x: x[0])
return [x[1] for x in r_list], [x[2] for x in r_list]
def append_sep(s_list):
r_list = []
for i, s in enumerate(s_list):
r_list.append(s)
r_list.append('[SEP_{0}]'.format(min(9, i)))
return r_list[:-1]
## print('convert into src/tgt format')
with open(os.path.join(args.output_dir, split_out+'.src'), 'w', encoding='utf-8') as f_src, open(os.path.join(args.output_dir, split_out+'.tgt'), 'w', encoding='utf-8') as f_tgt, open(os.path.join(args.output_dir, split_out+'.slv'), 'w', encoding='utf-8') as f_slv:
for src, tgt, lb in tqdm(zip(article_tk, summary_tk, label)):
# source
src_tokenized = [' '.join(s) for s in src]
if args.src_sep_token:
f_src.write(' '.join(append_sep(src_tokenized)))
else:
f_src.write(' '.join(src_tokenized))
f_src.write('\n')
# target (silver)
slv_tokenized = [s for s, extract_flag in zip(
src_tokenized, lb) if extract_flag]
f_slv.write(' [X_SEP] '.join(slv_tokenized))
f_slv.write('\n')
# target (gold)
f_tgt.write(' [X_SEP] '.join(
[' '.join(s) for s in tgt]))
f_tgt.write('\n')
The input should have been split by "<S_SEP>".
from unilm.
step 1: use corenlp to split the sentences
step 2: run BertTokenizer to obtain subword tokens
step 3: save the source text to *.src, and the target text to *.tgt
from unilm.
@donglixp , can you please provide details regarding how to run these 3 steps?
from unilm.
Thank you very much @donglixp .
from unilm.
@tahmedge Did you use above script? If yes, could you please share implementation of the same?
from unilm.
Related Issues (20)
- The influence of t (temperature) in the E5 Model paper HOT 4
- (BEIT-3) I can't find transformers.py HOT 3
- Kosmos-2.5 Dependencies Should Be Updated HOT 1
- The pretraining code and model of beit3
- TROCR model weight file cannot be downloaded.
- BEIT3 RuntimeError: Size mismatch when loading checkpoint for BEiT3ForRetrieval model
- Kosmos-2.5 - Deployment Challenges on a Windows 11 + RTX 3090 PC HOT 1
- [BEIT-3] error happens when I evaluate BEiT-3 finetuned model on VQAv2 HOT 4
- Missing Documentation
- Kosmos 2.5 dataset availability
- Kosmos-2.5 - Image-to-markdown generation for images outside the sample-set provided is almost entirely garbled - output markdown is completely unusable.
- Kosmos-2.5 - Python version 3.10.x (and any other confirmed working versions) should be mentioned as a requirement to deploy & infer the Kosmos-2.5 model
- Kosmos-2 batch modality and processing speed
- how to resume training for layoutlmv3-publaynet
- TextDiffuser - SD2.1 code and weights
- the problem when load Beats model
- kosmos-2.5 | trying to connect to `openaipublic.blob.core.windows.net`
- layoutlmv3 pretrain
- Compile LayoutLMv3 using Neuron SDK for AWS Inferentia (inf1)
- Beit-3 , clarification on classification task and associated pretrained weights
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from unilm.