Deep learning applications have seen substantial advancements with the advent of the Transformer architecture and its attention mechanism. Despite its success, Transformers face challenges in handling long sequences due to their inherent computational inefficiency. Several subquadratic-time architectures have been proposed to address this limitation, but they have not matched attention-based models’ performance on crucial tasks such as language processing. In this report, we identify the inability of these models to perform content-based reasoning as a key weakness and focus on Mamba, a novel neural network architecture that integrates selective structured state space models (SSMs) to address this limitation.

1. Report
2. Presentation

1. Fu, D. Y., Dao, T., Saab, K. K., Thomas, A.W., Rudra, A., and Ré, C. Hungry hungry hippos: Towards language modeling with state space models, 2023.
2. Gu, A., and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces, 2023.
3. Gu, A., Dao, T., Ermon, S., Rudra, A., and Re, C. Hippo: Recurrent memory with optimal polynomial projections, 2020.
4. Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces, 2022.
5. Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., and Ré, C. Combining recurrent, convolutional, and continuous-time models with linear state-space layers, 2021.
6. Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and Ré, C. Hyena hierarchy: Towards larger convolutional language models, 2023.
7. Smith, J. T. H., Warrington, A., and Linderman, S. W. Simplified state space layers for sequence modeling, 2023.
8. Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., and Metzler, D. Long range arena: A benchmark for efficient transformers, 2020.
9. Tay, Y., Dehghani, M., Bahri, D., and Metzler, D. Efficient transformers: A survey, 2022.
10. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023.

1. Sanidhya Singal
2. Aditya Gulati