Sparsifying Transformer Models with Differentiable Representation Pooling

A novel method to sparsify attention in the Transformer model by learning to select the most-informative token representations, thus leveraging the model's information bottleneck with twofold strength.