transformer architecture brought about a revolution sequence modeling with its introduction AttentionA mechanism by which models look at earlier inputs to prioritize relevant input data. However, computational cost increases drastically with sequence length, limiting the ability to scale Transformer-based models to extremely long contexts, such as those required for full-document understanding or genomic analysis.
The research community explored various approaches to solutions, such as efficient linear recurrent neural networks (RNN) and state space model (SSM) like Mamba-2These models provide fast, linear scaling by compressing the context to a fixed size, However, this fixed-size compression cannot adequately capture the rich information in very long sequences,
In two new letters, titans And inheritanceWe present an architectural and theoretical blueprint that combines the speed of RNNs with the accuracy of Transformers. TITANs is the specific architecture (tool), and MIRAS is the theoretical framework (blueprint) to generalize these approaches. Together, they advance the concept of test-time memorization, advancing the ability of AI models to retain long-term memory by incorporating more powerful “surprise” metrics (i.e., unexpected pieces of information) while the model is running and without dedicated offline retraining.
As demonstrated by Titans, the MIRAS framework offers a meaningful shift towards real-time optimization. Instead of compressing information in a static state, this architecture actively learns and updates its own parameters as the data streams. This important mechanism enables the model to quickly incorporate new, specific details into its original knowledge.
