Awesome SSM

This series will cover a bunch of posts about State Space Models, their extensions and applications.

Basics

• Mamba, Mamba2

Bidirectional

• Hydra

Attention Hybrids

• SSM-Transformer Hybrids covers:
  • An Empirical Study of Mamba-based Language Models
  • SAMBA Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling
  • Jamba A Hybrid Transformer-Mamba Language Model
  • Jamba 1.5 Hybrid Transformer-Mamba Models at Scale
  • Zamba A Compact 7B SSM Hybrid Model
  • Zamba2-small
  • Zamba2-mini
• Hymba
  • Hymba: A Hybrid-head Architecture for Small Language Models
  • instead of having Attention and Mamba running in alternating layers we run them in parallel in the same layer

Theory and Limitations

• Illusion of State in SSMs like Mamba covers:
  • The Expressive Capacity of State Space Models: A Formal Language Perspective
  • The Illusion of State in State-Space Models

With Graphs

• Graph Mamba: Towards Learning on Graphs with State Space Models
  • we leverage SSMs an alternative to Message Passing in Graph Neural Networks

Distillation

• Hohawk and Mamba in the Llama covers:

  • idea is to take an pretrained transformer and distill it into a SSM

Reinforcement Learning

• Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces
  • apply SSMs to Sequential Decision Making