Cached Transformers: Improving Transformers with Differentiable Memory Cachde

This work introduces a new Transformer model called Cached Transformer, which uses Gated Recurrent Cached (GRC) attention to extend the self-attention mechanism with a differentiable memory cache of tokens. GRC attention enables attending to both past and current tokens, increasing the receptive field of attention and allowing for exploring long-range dependencies. By utilizing a recurrent gating unit to continuously update the cache, our model achieves significant advancements in \textbf{six} language and vision tasks, including language modeling, machine translation, ListOPs, image classification, object detection, and instance segmentation. Furthermore, our approach surpasses previous memory-based techniques in tasks such as language modeling and displays the ability to be applied to a broader range of situations.

Authors

• Zhaoyang Zhang ARC Lab, Tencent PCG
• Wenqi Shao The Chinese University of Hong Kong
• Yixiao Ge Tencent Inc.
• Xiaogang Wang The Chinese University of Hong Kong
• Jinwei Gu The Chinese University of Hong Kong
• Ping Luo The University of Hong Kong

Keywords

• ML: Deep Learning Algorithms
• ML: Transfer, Domain Adaptation, Multi-Task Learning
• ML: Transparent, Interpretable, Explainable ML
• ML: Unsupervised & Self-Supervised Learning