State Space Models originated in control theory in the 1960s, and roughly 60 years later they are reshaping how AI processes long sequences. What started as a mathematical framework for engineering systems is now one of the most serious challenges to Transformer dominance in deep learning.
From Control Theory to Deep Learning Foundations
SSMs model sequences as linear dynamical systems defined by four parameter matrices: A, B, C, and D. For decades, these equations described physical systems like circuits and mechanical controls. The idea of turning them into general-purpose sequence learners took a long time to mature, mostly because the math did not map cleanly onto GPU hardware.
Key Milestones in State Space Model Evolution
2020: HiPPO Sets the Stage
HiPPO arrived in 2020 and tackled a fundamental problem: how to compress long continuous-time signals into a fixed-size state representation. This mathematical framework, called High-Order Polynomial Projection Operators, gave SSMs a principled way to preserve long-range dependencies across thousands of time steps. It was the theoretical bridge between classical control theory and modern deep learning.
2021: S4 Becomes the First Practical Deep Learning SSM
S4, introduced in 2021, became the first SSM that actually worked well in deep learning. It combined HiPPO's memory framework with structured matrix computations that GPUs could handle efficiently. S4 showed that linear-time sequence models could compete with Transformers on certain tasks, but adoption remained limited because the architecture was still rigid in how it processed different parts of a sequence.
December 2023: Mamba Brings Selective Processing
Albert Gu from Carnegie Mellon University and Tri Dao from Princeton University introduced Mamba in December 2023. The key innovation was selectivity: instead of treating every input token the same way, Mamba could choose which information to keep or discard from its hidden state. This made SSMs feel more like attention mechanisms, where not all tokens are equally important.
2024: Mamba-2 and Structured State Space Duality
Mamba-2 arrived in 2024 with a theoretical breakthrough called Structured State Space Duality, or SSD. SSD proved that every linear attention mechanism has an equivalent SSM representation. This was significant because it unified two research communities that had been working in parallel. Mamba-2 also better exploited modern GPU hardware, improving on the original implementation's efficiency.
2025-2026: The Hybrid Architecture Trend
By 2025 and into 2026, the conversation has shifted from 'SSMs versus Transformers' to 'how do we combine them.' Hybrid architectures are appearing across multiple domains. SSMs have been applied in natural language processing, speech recognition, vision, and time-series forecasting.
One concrete example is SlideMamba, a hybrid framework that fuses a graph neural network with a Mamba state-space branch for digital pathology analysis. Published in Scientific Reports in January 2026, SlideMamba outperformed the Transformer-based TransMIL on two tasks: achieving a PRAUC of 0.740 versus 0.390 on the OAK clinical trial dataset, and 0.969 versus 0.929 on a LUAD versus LUSC classification task. Meanwhile, models like RWKV demonstrate that linear-time inference with constant memory is achievable for text as well.
Why This Technology Matters Now
The core selling point of SSMs remains their complexity advantage. Transformers scale quadratically with sequence length, while SSMs achieve linear or near-linear complexity. But the industry is learning that raw efficiency is not enough. Transformers still excel at tasks requiring precise token-to-token comparisons, and SSMs still struggle with certain retrieval-heavy workloads. The evidence so far points to hybrids as the pragmatic path forward, not wholesale replacement.
Some bold claims about next-generation models achieving dramatic inference speedups over Transformers have circulated online, but these claims cannot be verified from available published research. The documented story is more measured: steady architectural refinement and smart hybrid designs.
The evolution from S4 to Mamba shows that AI architecture is far from settled. What do you think the dominant sequence modeling approach will look like five years from now: pure SSM, pure Transformer, or something we have not named yet?
Comments