What happens if texts from the dataset don't have equal lengths about recurrent-memory-transformer-pytorch HOT 4 CLOSED

@sentialx if everything in the second segment batch were the ignore index (zero in your case), it would be nan, but because there is [1, 2, 0, 0] you are still ok

from recurrent-memory-transformer-pytorch.

@sentialx hey Eryk! that should be handled if you pad your sequences with the ignore_index, which i have set to -1 but is configurable

https://github.com/lucidrains/recurrent-memory-transformer-pytorch/blob/main/recurrent_memory_transformer_pytorch/recurrent_memory_transformer.py#LL432C13-L432C25

from recurrent-memory-transformer-pytorch.

@lucidrains Thanks for your response, but what happens if a text has 4 segments and the second text in a batch has 2 segments? This means that the two remaining segments in the second batch slot are just filled with paddings to match the longest text, and there will be no predicted tokens, right?

To visualize this, here we have a tensor of shape (batch_size=2, segments=2, seq_len=5) and items are token ids:

[
  [
    [1, 2, 3, 4, 5]
    [1, 2, 3, 4, 0]
  ],
  [
    [1, 2, 0, 0]
    [0, 0, 0, 0] <- loss should be nan I think?
  ]
]

from recurrent-memory-transformer-pytorch.

@lucidrains one more question, did you have a chance to compare it with block recurrent?

from recurrent-memory-transformer-pytorch.