lucidrains / perceiver-ar-pytorch Goto Github PK

Hey @lucidrains I from classical ML and Speech Synthesis space, and I am sorta new to the NLP space and I have read this article with regards to decoding.
https://huggingface.co/blog/how-to-generate
I am wondering if the same techniques could be applied to character level embeddings? I am trying to find literature on the subject but I have hit a wall. If you know any resources a survey perhaps that would be helpful. It seems like the only thing that would work in a naive fashion here is greedy decoding

Hi, @lucidrains

Thanks for the implementation of Perceiver-AR model.
We conduct the experiments on pop music generation at: https://github.com/feizc/Perceiver-Music-Generation.
The results are encouraging, be grateful to you : )

Feel free to delete this issue if not useful...

import torch


def make_attention_mask(query_input: torch.Tensor, 
                        key_input: torch.Tensor, 
                        pairwise_fn=torch.multiply,
                        extra_batch_dims=0,
                        dtype=torch.bool):
    mask = pairwise_fn(query_input.unsqueeze(-1), key_input.unsqueeze(-2))
    mask = mask.unsqueeze(-3)
    for dim in range(extra_batch_dims):
        mask = mask.unsqueeze(dim)
    return mask.type(dtype)


def make_causal_mask(x, 
                     extra_batch_dims=0, 
                     dtype=torch.bool):
    idxs = torch.broadcast_to(torch.arange(x.shape[-1]), x.shape)
    return make_attention_mask(idxs, idxs, torch.greater_equal, extra_batch_dims=extra_batch_dims, dtype=dtype)


def combine_masks(*masks, dtype=torch.bool):
    masks_list = [m for m in masks if m is not None]
    if not masks_list:
        return None
    
    mask, *other_masks = masks_list
    for other_mask in other_masks:
        mask = torch.logical_and(mask, other_mask)
    
    return mask.type(dtype)

Hi Phil, would you mind explaining why do we need to rearange to (b, 1, n, d) here, I thought rotary_pos_emb should be (n, d) or (b, n, d):

please help me different convention with learned query version in perceiverAR

In your code you split the sequence into a prefix and smaller window and calculate the cross-attention with respect to it...

However, in the diagram of the method, the whole sequence is used for the V and K... Can you kindly confirm?

Thank you!

Thank you for providing this torch implementation of the Perceiver AR.

I was wondering if the following should be controlled by a flag?
In the original proposal it seems that the PerceiverAR only relies on the rotary positional encodings, rather than a combination of RoPE + learnable positional embedding.

I'd be happy to submit a PR if that's the case.

Thanks!

I think we can get away with having learnable null-latents in case we dont have prefix initially