2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]
Dec 24, 2024 · 42:45
Dan Fu (Together AI/UCSD) and Eugene Cheah (Featherless) survey post-transformer architectures, arguing that state-space models (SSMs) like Mamba and RWKV can match transformer quality while scaling subquadratically in sequence length, enabling efficient long-context and test-time compute. Fu explains how SSMs evolved from S4 to data-dependent selection mechanisms, while Cheah details RWKV's open-source development, including Q-RWKV6—a 32B model converted from Qwen 32B with linear attention that retains 76% MMLU after just days of training. They discuss hybrid models outperforming pure transformers or SSMs, and challenge RAG's necessity: fixed-state models like RWKV may handle up to 128K context for most enterprise workloads, while truly infinite context requires the model to decide what to forget. Hot takes: long context is overhyped for most use cases, but models that can 'watch forever' could excel in time-series and video generation.
