Memory Mosaics

Memory Mosaics are networks of associative memories working in concert to achieve a prediction task of interest. Like transformers, memory mosaics possess compositional capabilities and in-context learning capabilities. Unlike transformers, memory mosaics achieve these capabilities in comparatively transparent way (“predictive disentanglement”). We illustrate these capabilities on a toy example and also show that memory mosaics perform as well or better than transformers on medium-scale language modeling tasks.

Authors

• Jianyu Zhang
• Niklas Nolte
• Ranajoy Sadhukhan
• Beidi Chen
• Léon Bottou

Keywords

• predictive disentanglement
• Associative memory
• language model