This series will cover a bunch of posts about State Space Models, their extensions and applications.

Basics Link to heading

• Mamba, Mamba2

Bidirectional Link to heading

• Hydra

Theory and Limitations Link to heading

• The Expressive Capacity of State Space Models: A Formal Language Perspective
  • look at SSMs from a lens of regular languages
• The Illusion of State in State-Space Models
  • look at the limitations of SSMs, especially when it comes to tracking state in Chess, Code and other domains

With Graphs Link to heading

• Graph Mamba: Towards Learning on Graphs with State Space Models
  • we leverage SSMs an alternative to Message Passing in Graph Neural Networks

Distillation Link to heading

• Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models
  • idea is to take an pretrained transformer and distill it into a SSM

Reinforcement Learning Link to heading

• Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces
  • apply SSMs to Sequential Decision Making