Together AI

⚡️ Reverse Engineering OpenAI's Training Data — Pratyush Maini, Datology

Feb 10, 2026 · 27:02

Pratyush Maini from Datology reveals how the seahorse emoji question exposes that frontier models like GPT-4.1, GPT-5, and OLMo 3.1 now exhibit self-correction loops—a sign that reasoning traces are baked into mid-training data, not just post-training. By tracking response length surges four months after o1's release and corroborating with OLMo's data, Maini argues that foundation models must contain core capabilities like self-reflection, reinforcing the 'Fine Tuner's Fallacy': you cannot simply fine-tune for a capability; it must be in pre-training. He also highlights Datology's BeyondWeb project, which uses a source-rephrasing paradigm to transform internet data at trillion-token scale, achieving Nemotron-level performance in 2.7x less compute—making specialized pre-training accessible for enterprises. The episode concludes that 2026-2027 will see a shift toward domain-specific pre-training, as the cost amortizes when a smaller pre-trained model outperforms a larger fine-tuned one.

2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]

Dec 24, 2024 · 42:45

Dan Fu (Together AI/UCSD) and Eugene Cheah (Featherless) survey post-transformer architectures, arguing that state-space models (SSMs) like Mamba and RWKV can match transformer quality while scaling subquadratically in sequence length, enabling efficient long-context and test-time compute. Fu explains how SSMs evolved from S4 to data-dependent selection mechanisms, while Cheah details RWKV's open-source development, including Q-RWKV6—a 32B model converted from Qwen 32B with linear attention that retains 76% MMLU after just days of training. They discuss hybrid models outperforming pure transformers or SSMs, and challenge RAG's necessity: fixed-state models like RWKV may handle up to 128K context for most enterprise workloads, while truly infinite context requires the model to decide what to forget. Hot takes: long context is overhyped for most use cases, but models that can 'watch forever' could excel in time-series and video generation.

Building an open AI company - with Ce and Vipul of Together AI

Feb 8, 2024 · 1:15:18

Together AI co-founders Vipul Ved Prakash and Ce Zhang explain why openness is core to their mission, detailing their journey from Apple (Vipul) and Stanford research (Ce) to building an open AI platform. They discuss RedPajama’s evolution into a modular dataset with 40 quality signals, the need for 5,000 tokens/second inference speed, and their investment in state space models like Mamba and Hyena as alternatives to transformers. The company, which runs 7,000-8,000 GPUs (mostly H100s), sees training as a larger workload than inference, with fine-tuning driving top models. They advocate for independent inference benchmarks, publish open research like FlashAttention, and keep some software proprietary. With 38 employees and 45% researchers, they are hiring across the stack, from CUDA to DevOps.