Apple intelligence

[State of Context Engineering] Agentic RAG, Context Rot, MCP, Subagents — Nina Lopatina, Contextual

Dec 31, 2025 · 26:48

Nina Lopatina of Contextual AI discusses the state of context engineering, arguing that while the field is still in prototyping, 2025 will see true scale with full-system design patterns. She notes that agentic RAG—especially query reformulation into subqueries—has become the baseline, dramatically improving retrieval. Context rot is widely cited but industry benchmarks at real scale (100k+ documents, billions of tokens) remain rare. MCP is both a driver and a flaw: giant JSON tool definitions stuff the context window, yet MCP servers enable rapid prototyping before optimizing down to direct API calls. Sub-agents with turn limits are critical—unlimited agency degrades performance and causes hallucinations. Instruction-following re-rankers are essential for scaling retrieval across massive databases, providing more recall upfront and precision in the final context. Benchmarks are being saturated faster than ever: a Princeton benchmark from October was saturated by Claude Code in weeks, with solutions so good the gold dataset had errors. Lopatina also shares a lesson from using ChatGPT as a snowboarding coach—after several months of training for a 25-lap mogul race, she had to close…

Best of 2024: Synthetic Data / Smol Models, Loubna Ben Allal, HuggingFace [LS Live! @ NeurIPS 2024]

Dec 24, 2024 · 28:08

Loubna Ben Allal of Hugging Face explains how synthetic data has become ubiquitous in LLM pipelines, moving from post-training to pre-training, with examples like Cosmopedia’s 30B-token synthetic dataset and NVIDIA’s 1.9T-token Nemotron CC. She addresses model collapse fears, showing that web dumps after ChatGPT’s release actually yield better models, and emphasizes diversity through prompt seeds and webpage extracts. For filtering, FineWeb-Edu and DCLM use LLMs to rate educational content, achieving top benchmark scores. In post-training, she highlights AgentInstruct, Tülu3 with PersonaHub, and Cohere’s multilingual arbitrage using multiple teachers. On small models, SmolM2 (1.7B) outperforms Llama 1B and Qwen 2.5 after 11T tokens of pre-training, and on-device inference via frameworks like llama.cpp enables privacy-preserving use cases such as text extraction and structured generation. She predicts a return to fine-tuning specialized small models over costly prompt engineering.