Implementation of gMLP from Pay Attention to MLPs, an all-MLP replacement for Transformers, in tensorflow
model = tf.keras.Sequential([
gMLPLayer(),
gMLPLayer(dropout_rate=0.05),
Flatten(),
Dense(3)
])
@misc{liu2021pay,
title = {Pay Attention to MLPs},
author = {Hanxiao Liu and Zihang Dai and David R. So and Quoc V. Le},
year = {2021},
eprint = {2105.08050},
archivePrefix = {arXiv},
primaryClass = {cs.LG}
}