jingtaozhan / repbert-index Goto Github PK

Hello,

I'm trying to reimplement the model and I found that the situation between train and inference is different.
In the train.py, the model feed the input like format as (query, doc) but in the precompute.py, the model feed input as (query) and (doc) seperately.
According to your paper, embed(query) and embed(doc) is right I think.
Am i miss something?

Thank you

Thanks a lot for sharing this work and I have successfully reproduced the MRR@10 metrics on the MS Marco Passage ranking task using your published checkpoint. I'm wondering if you are planning on sharing the checkpoint or code on how to reproduce the new LTRe model which achieves better MRR accuracy after training. https://arxiv.org/abs/2010.10469

Again, thanks a lot for sharing this work.

请问一下你的基本运行环境是怎么样的，比如torch和transformers的版本？谢谢。

Hello, I just read the complete code of the RepBERT, and there seems to be a small bug in "train.py". In lines 91 and 93, the "_" is useless, right?

for epoch_idx, _ in enumerate(train_iterator):
  epoch_iterator = tqdm(train_dataloader, desc="Iteration")
    for step, (batch, _, _) in enumerate(epoch_iterator):

您好，我想用这个模型处理一些中文任务。

我主要是想知道您是如何训练bert的。我手头有一个1m的语料库。

此外，转中文的话我还需要注意哪些问题。

谢谢

Hello,

Thank you for your good work.
How much do you think it takes approximately to precompute all the memmaps for msmarco collection on one RTX-2080- 8GB gpu? It seems it takes several days... I want to make sure I am on the correct path.

Hello, I notice you try to mask the attention between the query and document. However, it seems to be a problem when the input sequence is padded.

def _mask_both_directions(valid_mask, token_type_ids):
    assert valid_mask.dim() == 2
    attention_mask = valid_mask[:, None, :]

    type_attention_mask = torch.abs(token_type_ids[:, :, None] - token_type_ids[:, None, :])
    attention_mask = attention_mask - type_attention_mask
    attention_mask = torch.clamp(attention_mask, 0, None)
    return attention_mask

e.g., if "token_type_ids" equals [0, 0, 0, 1, 1] (without padding),
then the "type_attention_mask" will be
|0 0 0 1 1|
|0 0 0 1 1|
|0 0 0 1 1|
|1 1 1 0 0|
|1 1 1 0 0|
and "attention_mask" will be
|1 1 1 0 0|
|1 1 1 0 0|
|1 1 1 0 0|
|0 0 0 1 1|
|0 0 0 1 1|
it is correct.
However, if "token_type_ids" equals [0, 0, 0, 1, 1, 0] (with padding, max_length=6),
then the "type_attention_mask" will be
|0 0 0 1 1 0|
|0 0 0 1 1 0|
|0 0 0 1 1 0|
|1 1 1 0 0 1|
|1 1 1 0 0 1|
|0 0 0 1 1 0|
and "attention_mask" will be
|1 1 1 0 0 0|
|1 1 1 0 0 0|
|1 1 1 0 0 0|
|0 0 0 1 1 0|
|0 0 0 1 1 0|
|1 1 1 0 0 0|
it is not the expected attention mask, right?