Questions about the pre-trained language model about encodec HOT 4 OPEN

Thanks for your reply, but I want to ask what is the exact usage of this transformer module, how it can reduce bandwidth?

The transformer is trained a posteriori using a trained encoder / quantizer / decoder. The language model is not pretrained, it is only trained on the task of modeling the tokens from the underlying quantizer.

from encodec.

The transformer is not trained in streaming mode, although we use some tricks to make it more compatible with streaming, e.g. we use random initial offset in the positional embedding and we limit the receptive field into the past. Then it is used in streaming mode, in particular in the decoder, as you must be able to decode the current token before you can make sense of the following bits.

from encodec.

The transformer is trained a posteriori using a trained encoder / quantizer / decoder. The language model is not pretrained, it is only trained on the task of modeling the tokens from the underlying quantizer.

from encodec.

If I can answer your question, it comes from Shannon's source coding theorem: given an alphabet of symbols, you can choose shorter codes for more likely symbols to reduce the overall length of the transmitted message, and thus bandwidth. The Transformer is used to estimate dynamically this code indices' probability in order to change the way they are encoded.

Also, I have another question: does the transformer is also trained in streaming mode (just like inference, with as many forwards as the encoded sequence length) or in classic seq2seq mode (with a single forward path)?

from encodec.