This series will cover a bunch of posts about State Space Models, their extensions and applications.

Basics Link to heading

• Mamba, Mamba2

Bidirectional Link to heading

• Hydra

Attention Hybrids Link to heading

• SSM-Transformer Hybrids covers:
  • An Empirical Study of Mamba-based Language Models
  • SAMBA Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling
  • Jamba A Hybrid Transformer-Mamba Language Model
  • Jamba 1.5 Hybrid Transformer-Mamba Models at Scale
  • Zamba A Compact 7B SSM Hybrid Model
  • Zamba2-small
  • Zamba2-mini

Theory and Limitations Link to heading

• Illusion of State in SSMs like Mamba covers:
  • The Expressive Capacity of State Space Models: A Formal Language Perspective
  • The Illusion of State in State-Space Models

With Graphs Link to heading

• Graph Mamba: Towards Learning on Graphs with State Space Models
  • we leverage SSMs an alternative to Message Passing in Graph Neural Networks

Distillation Link to heading

• Hohawk and Mamba in the Llama covers:

  • idea is to take an pretrained transformer and distill it into a SSM

Reinforcement Learning Link to heading

• Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces
  • apply SSMs to Sequential Decision Making