AttentionLite: Towards Efficient Self-Attention Models for Vision

Intel Labs has created a novel framework for producing a class of parameter- and compute-efficient models called AttentionLite, which leverages recent advances in self-attention as a substitute for convolutions.