Kwaai Innovative Architectures Research Lab

Goals of the Group

Develop a light weight language model implementaion of Mamba (or alternatives) for use in the Kwaai Personal Operating Sytem.

Useful Reads

• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Graph Mamba: Towards Learning on Graphs with State Space Models
• Transfomers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
• MambaByte: Token-free Selective State Space Model
• Jamba: Hybrid Transfomer-Mamba Language Model
• Black Mamba: Mixture of Experts Mamba