# [State of Context Engineering] Agentic RAG, Context Rot, MCP, Subagents — Nina Lopatina, Contextual Latent Space · 2025-12-31 Nina Lopatina of Contextual AI discusses the state of context engineering, arguing that while the field is still in prototyping, 2025 will see true scale with full-system design patterns. She notes that agentic RAG—especially query reformulation into subqueries—has become the baseline, dramatically improving retrieval. Context rot is widely cited but industry benchmarks at real scale (100k+ documents, billions of tokens) remain rare. MCP is both a driver and a flaw: giant JSON tool definitions stuff the context window, yet MCP servers enable rapid prototyping before optimizing down to direct API calls. Sub-agents with turn limits are critical—unlimited agency degrades performance and causes hallucinations. Instruction-following re-rankers are essential for scaling retrieval across massive databases, providing more recall upfront and precision in the final context. Benchmarks are being saturated faster than ever: a Princeton benchmark from October was saturated by Claude Code in weeks, with solutions so good the gold dataset had errors. Lopatina also shares a lesson from using ChatGPT as a snowboarding coach—after several months of training for a 25-lap mogul race, she had to close… ## Questions this episode answers ### Is agentic RAG really better than traditional RAG for context engineering? Nina Lopatina argues that the debate over agentic RAG versus traditional RAG is largely settled: even the simple step of query reformulation into sub-queries improves retrieval performance so dramatically that agentic RAG is now the baseline for context engineering systems. [4:47](https://addtry.com/c24663ed-8f3b-4cb8-8419-50c1a6e479f6?t=287000) ### What are the benefits and drawbacks of using MCP for context engineering? Nina Lopatina explains that MCP has driven context engineering by enabling rapid prototyping and quick tool combination, but it also leads to context rot because the large JSON tool definitions stuff the context window. She recommends using MCP for early prototyping, then moving to direct API calls to reduce complexity and dependency. [14:20](https://addtry.com/c24663ed-8f3b-4cb8-8419-50c1a6e479f6?t=860000) ### How did AI systems perform compared to a human in the Retail Universe hackathon? During the Retail Universe hackathon with around 100,000 documents, the human benchmark scored 23 points. Nina Lopatina’s team, using a dynamic agent with explicit turn limits, scored about 25. The winning team, using Mix Spread and Claude, scored roughly 29, indicating that the best systems exceeded human-level performance. [6:20](https://addtry.com/c24663ed-8f3b-4cb8-8419-50c1a6e479f6?t=380000) ### How quickly was the Princeton HAL benchmark for agentic tasks saturated? Nina Lopatina notes that a Princeton HAL benchmark for recreating a research paper, released in October, was saturated by Claude Code earlier that same week (around mid-December). The solutions were so advanced that human evaluators had to be used, and errors were found in the original gold dataset. [10:30](https://addtry.com/c24663ed-8f3b-4cb8-8419-50c1a6e479f6?t=630000) ## Key moments - **[0:00] NeurIPS Kickoff** - [1:40] OpenRouter's state of AI survey reveals small language models (<15B) market share is trending down over time. - [3:09] Contextual released the first instruction-following re-ranker in March to improve retrieval precision and combat context rot. - **[4:46] Agentic RAG** - [4:57] Agentic RAG with query reformulation into subqueries improved performance so dramatically it became the new baseline, says Nina Lopatina. - [6:37] Nina Lopatina found that sub-agents in a hackathon over 100,000 documents needed explicit turn limits to prevent endless validation loops. - **[8:22] Hackathon** - **[9:17] Context Engineering** - [9:20] Context engineering took hold in mid-2024 and remains in a prototyping stage with many design patterns but no uniform architecture. - [10:55] Claude Code saturated the Princeton HAL benchmark for recreating research papers just two months after its October release. - [13:23] Context rot is cited in every blog, but industry benchmarks at real scale (100k+ documents, billions of tokens) are still rare. - **[13:47] Context Rot & MCP** - [14:26] MCP's giant JSON tool definitions quickly cause context rot, but its servers accelerate prototyping before optimizing to direct API calls. - **[17:36] Optimizing Context** - [18:36] Agentic context engineering (ACE) makes smaller tweaks to prompts rather than rewriting from scratch to avoid performance drop-offs. - **[21:03] Embodied Evals** - [21:03] Nina Lopatina embodied-evaled ChatGPT as a snowboarding coach for a 25-lap mogul race, but had to restart the window because the model lost training context. - [22:42] Contextual's platform achieved state-of-the-art human evals for test code generation by using multimodal ingestion and hybrid search. - **[23:59] Looking Ahead** - [24:16] Nina predicts that in 2026 context engineering will shift its focus from individual components to full-system design patterns. - [25:12] Nina's first NeurIPS in 2016 Barcelona was smaller and lacked industry parties; now the conference is much larger and more commercial. ## Speakers - **Nina Lopatina** (guest) ## Topics Agent Infrastructure, Benchmarks ## Mentioned Anthropic (company), Arize (company), Cloudflare (company), Contextual (company), Manus (company), NVIDIA (company), Turing (company), Apple intelligence (product), ChatGPT (product), Chroma (product), Claude (product), Cursor (product), Gemini (product), LM Arena (product), MCP (product), OpenRouter (product) ## Transcript ### NeurIPS Kickoff **Host** [0:03] Light in space time 2025. Break ups. Light in space time 2025. We are here back, uh, with Nina Lopatina, welcome, from Contextual. **Nina Lopatina** [0:16] Thanks. **Host** [0:17] We're gonna talk about the state of context engineering in general, but, uh, one thing I wanted to also give people a sense who are not here of the NeurIPS flavor discussion and all that. We were talking about, like... I was asking you which are the best NeurIPS after parties, and you said Makor, and what's the other one? **Nina Lopatina** [0:32] Turing and NVIDIA. **Host** [0:33] Turing and NVIDIA, where there was a really good fireside chat discussion. **Nina Lopatina** [0:35] Mm-hmm. **Host** [0:36] Which is rare 'cause I, I often tend to shy away from fireside chats. **Nina Lopatina** [0:40] Mm-hmm. **Host** [0:40] Uh, but apparently the guest was Yejin, uh, Choi, who is a very well-known figure around these parts. What was it about? **Nina Lopatina** [0:46] Yeah. It was about, uh, kind of the overall state of AI. Uh, since she's at NVIDIA, they talked about, a lot about scaling laws and the future of AI. I think Jonathan Sidar, the, uh, CEO of Turing who was interviewing her, was just, uh, great at asking fun questions, as was the audience. I would say, uh, the, the last question there was kind of perfect. Uh, so she focuses on small language models and, uh, the question was, you know, "Wouldn't your research cause NVIDIA to, to lose a lot of market cap?" And in fact she had emailed her, uh, research to Jensen Huang before joining and he was like, "Yeah, that's great." And I think that's- **Host** [1:24] He doesn't care. **Nina Lopatina** [1:25] Yeah. **Host** [1:25] He's really in the world right now. **Nina Lopatina** [1:26] Well, no, but actually the implication is that, uh, if you have smaller language models you can do more, you can have them run on phones, and so you actually could end up using more compute to, uh, to power more and more small language models to, to do more so... **Host** [1:40] Yeah. Um, I would say though, uh, so also at NeurIPS the OpenRouter team released their state of AI survey based on OpenRouter and they had this very interesting chart on the adoption of small versus medium versus large language models. **Nina Lopatina** [1:54] Mm-hmm. **Host** [1:54] So the cutoff was small was less than 15 billion and medium was 15 to 70 and large is anything above 70. And you could see the market share of small l- models trending down over time in practice. **Nina Lopatina** [2:06] Really? **Host** [2:06] And the market share of medium trending up and then large staying the same. **Nina Lopatina** [2:10] Mm-hmm. **Host** [2:10] Yeah. And this is open models only of course. **Nina Lopatina** [2:12] Mm-hmm. **Host** [2:13] And obviously the closed models have gone up and up and up so... I don't, I don't... It's just a, just a data point. **Nina Lopatina** [2:18] Yeah. **Host** [2:18] You know, I think, I think it was one of those things where like Apple intelligence I, I think it's widely in- in- acknowledged to be a failure this year and they were kind of the champions of the small models, uh, on device movements, uh, that really needed to work and it did not work. Gemini launched, uh, Gemini Nano in Chrome this year. I haven't used it. I don't know if you've used it. Yeah, basically like the small model hope and dream I think it's, it's still in like very small use cases and not rolled out and I don't know what needs to happen for it to roll out. **Nina Lopatina** [2:48] Yeah. I would say that makes a lot of sense for general purpose models. I think for, uh, other component models like let's say like for a re-ranker due to latency constraints- **Host** [2:57] Of course **Nina Lopatina** [2:58] ... smaller is better is what I've heard from other developers, but I can see that being a trend more generally. **Host** [3:03] Yeah. You guys had, uh, a lot of re-ranker work last year. I don't know, is, is that like as much of a focus this year for Contextual? **Nina Lopatina** [3:09] We released the first instruction following re-ranker in March. **Host** [3:13] Okay. **Nina Lopatina** [3:13] And we updated it recently so it's something that I think we'll keep digging to keep up with the state of the art 'cause it is really a key function within context engineering. So for example, when you're reasoning over larger and larger databases you wanna be able to have more recall in those initial retrievals but you want more precision for what you're actually put- putting into the context window for fear of context rot and, you know, poor performance and so that re-ranker can really help you narrow down that initial retrieval. **Host** [3:46] I did a brief comment that it's weird that, that that was the first instruction following re-ranker because well don't you always want it to be instruction following? And uh, the simple answer from, uh, that I've had I've, I've literally asked the search firms this, the startups like, um, you know, those people who know will know and they, they're all like, "Well, you know, to be instruction following you have to have, uh, the, a larger model and that affects our latency budget." And I'm like, I, I don't think that's the case like like la- latency for, for smaller models are pretty good these days. I don't know. **Nina Lopatina** [4:19] Yeah. So actually that is like the biggest complaint I've heard from developers about our re-ranker- **Host** [4:24] Okay **Nina Lopatina** [4:24] ... the, the latency. **Host** [4:25] Latency yeah. **Nina Lopatina** [4:25] But what we're using it for increasingly is for dynamic agents and there you're- **Host** [4:30] So that's insensitive right? **Nina Lopatina** [4:32] Yeah. **Host** [4:32] Just take however long you wanna take. Yeah. **Nina Lopatina** [4:33] Exactly. **Host** [4:34] Yeah. That's, that's fair. I'll get to right to context engineering. I think the other thing that people are... There was a big topic of conversation this year was quote unquote "the, uh, death of normal RAG and the rise of agentic RAG." ### Agentic RAG **Host** [4:47] Do you agree? Uh, is it o- is the debate itself overrated because obviously you use whatever in the right context, right? Like it depends. **Nina Lopatina** [4:55] Mm-hmm. **Host** [4:55] I don't know. Is, is this a meaningful debate? **Nina Lopatina** [4:57] I mean I think the debate is not so meaningful. I think like progress is, is meaningful but I think to me it's also like maybe somewhat a decided debate like agentic RAG is just generally better than RAG even that initial incremental step of making that, uh, doing query reformulation so when you receive that initial query being able to break it down into sub-queries so that you can better match those queries to documents that you might wanna retrieve and then combine that for the retrieval even that step improves performance so dramatically that, you know, that's kind of the- **Host** [5:31] Yeah **Nina Lopatina** [5:31] ... that became the new baseline. **Host** [5:33] Yeah. Um, that, that is obviously so helpful and you can farm it out and parallelize and then re- regroup it. Uh, so this year the work that I did on Sweetgrab, uh, did that like so we, we had like a fast context model where it was trained. Uh, I think a couple innovations I think was really interesting one, uh, train for massively or much more parallel than normal tool calling for searching so you can sort of, uh, normal, normal parallelism is like one to two maybe maximum four. We train the baseline to be six, uh, parallel searches at a time and goes up, goes up to eight. And then also limited agency, that you don't want your agentic search to run forever. You, you do want it to terminate at some point and return the answer. So, uh, incentivizing the RL to do that I think was helpful, easy, and actually it scaled very well. **Nina Lopatina** [6:20] Yes. And actually we have found that, uh, having turn limits and limits on the sub-agents', uh, checking and validating their work is super important. We recently did, uh, my, uh, myself and a member of my team recently participated in a context engineering hackathon. **Host** [6:37] Yeah, tell us about it. **Nina Lopatina** [6:38] Yeah. So Brian Bischoff and Hamill Hussein hosted it in San Francisco last month, mid-November, and they had about, uh, just under 100,000 documents, so PDFs all in the retail space, so it's called Retail Universe. And they had log files, uh, tables, giant CSVs, CRK files, and we used our dynamic agent to answer really challenging queries in this and generate structured output. And some of the very early steps we noticed is, you know, with a data set that large, that agent will wanna take... That sub-agent, let's say the unstructured retrieval or structured retrieval sub-agent will wanna take so many turns to make sure they've, you know, looked under every rock in the data set, and we actually don't want that. That data set is pretty large. And then also it turns out the sub-agent will want to check its work over and over and over. So, um, so that's kind of like something that we noticed with large scale, and so kind of similar to what you found, like you don't want unlimited, you know, whether you're enforcing that within like the RL reward structure or with explicit instructions in the system prompt. **Host** [7:39] Yeah. I, I would say very much this year is kind of the year of the sub-agent for me in terms of like there are people are use- using kind of constrained agents to do very specific things, and, uh, sometimes being too general actually is like an anti-pattern because, well, it doesn't go very far on its own, or it's not very reliable, or it doesn't have the tools that it needs, or blah, blah, blah, blah, blah. Right? Uh, and sub-agents make sense. Uh, also, like you also happen to be able to fine-tune a model towards that specific task, so you can also hill climb very easily as well. And, and, and so I think to me that is like all the right elements of AI engineering that I want to see people do more of, and people just needed a term for it, and I guess I think people have settled on sub-agents as, as a term. Just briefly staying on the hackathon, are there any other alternative approaches that you found interesting that you wanna shout out? ### Hackathon **Nina Lopatina** [8:28] I actually have a blog I'll write, uh, soon about our experience and our- **Host** [8:31] Mouth blog it. I love it. **Nina Lopatina** [8:33] And so I think the leading team was using Mix Spread and Claude. Second place was using Cursor. **Host** [8:40] Mix Spread? **Nina Lopatina** [8:41] Yeah. **Host** [8:41] I've, uh, seen that a couple times. **Nina Lopatina** [8:43] Yeah. I think they also have a re-ranker and some other open source models. **Host** [8:46] Okay. **Nina Lopatina** [8:46] Um- **Host** [8:47] Interesting name. **Nina Lopatina** [8:48] Yeah. And I forget what the other team was. And, and there was also a human doing the challenge, um, so there was a human benchmark. **Host** [8:55] Oh, yeah. How'd the human do? **Nina Lopatina** [8:57] Uh, I think they got 23 points. We got, I think, about 25 and the- **Host** [9:01] Nice **Nina Lopatina** [9:01] ... winners had about 29, so it was like very close. **Host** [9:03] Okay, superhuman. Yeah. **Nina Lopatina** [9:04] Yeah. **Host** [9:05] Yeah, yeah, yeah. Uh, very cool. Okay, so let's get right into the meat, uh, which is context engineering. This very, very big year for context engineering. Your company is Contextual, which like could not be more on the nose. How do you describe this year in context engineering? ### Context Engineering **Nina Lopatina** [9:20] It's been a very fast year because context engineering really took hold six months ago, and that actually feels like a year. And so I think one thing that stands out to me is that there are a lot of design patterns that, um, are kind of bubbling up, but there isn't like a uniform design that folks are using as the, as the architecture. And I think there's a lot of optimization and efficiencies to gain. So I think with a lot of new development you kind of start by, you know, letting the agent use as many tokens as it wants, and then later you figure out how to constrain that and how to optimize it, uh, let's say like with, uh, key value caching or other approaches that can help, um, really scale the technology. So I think it's kind of, to me, maybe more in a prototyping stage, and I'm expecting next year we'll really see scale for context engineering. **Host** [10:11] What does scale look like? What will we be able to do end of next year that we cannot do this year or that we're not doing this year? **Nina Lopatina** [10:17] I think the kinds of tasks that we'll be able to solve are going to increase. So, uh, I think for example we're already seeing the start of that. There was in the HAL benchmarks there was one about, uh- **Host** [10:30] Is this the ch- the Chroma piece? **Nina Lopatina** [10:32] Uh, no. This is from Princeton. **Host** [10:34] Okay. No, then I, I'm not familiar with it. **Nina Lopatina** [10:37] Uh, HAL. I forget now what it stands for, but it's a set of benchmarks for, uh, really evaluating longer running agentic tasks, and in this case there was one where they were evaluating recreating a research paper, and that benchmark came out in October, and it was saturated earlier this week- **Host** [10:54] Oh, geez **Nina Lopatina** [10:55] ... by Claude Code. **Host** [10:56] Oh, geez. **Nina Lopatina** [10:56] Yeah. So and in fact, uh, they needed to have humans run the eval- do the evaluation because the solution was like somewhat like maybe a different approach than a human would take, somewhat superhuman. You know, the com- the common pattern you see now where actually like the, the gold data set has some errors in it, and so the model- **Host** [11:16] Love it **Nina Lopatina** [11:16] ... isn't correct. **Host** [11:17] Yeah. **Nina Lopatina** [11:17] Um, and it marked it wrong. **Host** [11:18] Because like, because if it gets 100 then you're like, "Oh, well there's something's wrong." Like it's a, it's a canary for- **Nina Lopatina** [11:23] Mm **Host** [11:23] ... well, something's actually wrong. Yeah. **Nina Lopatina** [11:25] Yeah. **Host** [11:25] So we should do it on purpose. **Nina Lopatina** [11:27] Yeah. And I think we'll just see these benchmarks, you know, these new, you know, really well thought out and really challenging benchmarks come out and then very quickly be, uh, saturated. **Host** [11:38] Yeah. **Nina Lopatina** [11:38] And we'll continue to see that, I think, for more and more challenging tasks. **Host** [11:41] Do you find in, like just general DevRel work and marketing and just like leadership of a category, like do you find it useful to maintain a benchmark or like to, to have like, oh, this is the contextual benchmark that everyone should, should adopt? I, I struggle with this because obviously a lot of benchmarks come from research and not industry, but I feel like industry should have a role. **Nina Lopatina** [12:02] Yeah. Actually, I mean, we've been using that data set from the Uh, hackathon I described earlier- **Host** [12:07] Yeah **Nina Lopatina** [12:07] ... to a benchmark somewhat. I think it's a, it's a really interesting data set because most benchmarks use a very small set of data to train or to inference about, and this actually requires reasoning over- **Host** [12:20] Yeah, this will never fit in a context window, right? **Nina Lopatina** [12:23] Yes. **Host** [12:23] So. **Nina Lopatina** [12:23] Yeah. So I think- **Host** [12:24] Do you know how many tokens it is or... You, you said it's 100,000 documents. **Nina Lopatina** [12:28] Yeah. **Host** [12:29] But I don't know how many tokens that translates to. **Nina Lopatina** [12:31] Oh. **Host** [12:31] I'd say, like, about 1,000 each, 4,000 each? **Nina Lopatina** [12:35] Probably more. **Host** [12:35] Probably more, okay. So that's many billions. **Nina Lopatina** [12:38] Yes. **Host** [12:38] Yeah. Okay, cool. **Nina Lopatina** [12:39] Yes. So I think, um, I would love to see benchmarks. Uh, yeah, I would love to see more industry benchmarks because those would help us actually evaluate at the scale and not have a toy example as, you know, many benchmarks are. **Host** [12:51] Yeah, amazing. And you know, just like the, the so the lore of context engineering, obviously we got a shout-out to Dex who, uh, did a couple great talks this year, maybe, maybe three. I think also Drew Bernig als- written a lot about the failure modes of context engineering. What's sticking with the people that you talk to, right? Like context rot is, it's a, it's a well-established term, shout-out to Chroma. Anything else? Context poisoning I haven't heard, heard as much, right? So that, that term is not sticking. What, what else is, like, the topic of conversation that everyone should know? **Nina Lopatina** [13:23] Yeah, I mean, I think I see context rot cited in every blog about- **Host** [13:26] Yeah. Uh, yeah, and I, I always call it my L on that. Like I told Jeff, I was like, "Are you sure this needs to be written because everyone knows this?" And he's like, "No, everyone everyone doesn't know this." **Nina Lopatina** [13:37] Well, exactly. **Host** [13:37] It was, like, obvious to us, but- **Nina Lopatina** [13:38] Yes **Host** [13:39] ... you know. **Nina Lopatina** [13:39] Yes. I think it's very intuitive, but I think actually having the metrics and results to show for- **Host** [13:45] Yes, he did the work. **Nina Lopatina** [13:46] Yes, exactly. **Host** [13:47] He did the work. ### Context Rot & MCP **Nina Lopatina** [13:47] Yeah. **Host** [13:47] You know, like a lot of people have intuitions just based on using a model, but actually if you can put a model saying, put a number saying like, "Well, okay, you know, of this million token context at 700,000 tokens, your retrieval is actually, like, 30%." **Nina Lopatina** [14:00] Exactly, yeah. So now you can compare it to other performance gaps and kind of see what's having a bigger impact. I think, uh, Anthropic has had some really great blogs in this space. I would say, uh, there's one on some design and architecture choices that they put out that was really interesting, uh, fairly, fairly early. I would say, you know, it didn't come out this year, but I think MCP has been a huge driver of context engineering. **Host** [14:26] Driver and also a flaw, I would say as well. Let's talk about it. **Nina Lopatina** [14:32] Yeah. **Host** [14:33] Because MCP is this giant JSON thing upfront. You're stuffing in the descriptions of all the tools, and so it's very quickly you get into straight, straight up context rot when you have, like, 10 tools, especially if the tools are fat. **Nina Lopatina** [14:47] Yeah. So there's, uh, been some really interesting work on, uh, some really interesting blogs on tool use. Uh, Manus had one that was kind of more general but had some best practices around tools, and Anthropic has written more on, on tool use patterns. So- **Host** [15:00] Cloudflare. **Nina Lopatina** [15:01] Yeah. **Host** [15:02] That's the other one? Yeah. **Nina Lopatina** [15:03] Yeah. We actually, uh, funny enough, going back to our re-ranker, I actually set up, uh, just a prototype of selecting which MCP servers to use. Uh, being able to select those servers is already, like, a context challenge 'cause there's so many of them. **Host** [15:18] It's a sub-agent. **Nina Lopatina** [15:19] Yeah. **Host** [15:20] Interesting. Is that something people are very excited about? They're deploying at scale, you see a lot of traction? **Nina Lopatina** [15:26] I think similarly... So I think, yes, we've definitely seen a lot of great use for it. We have our dynamic agent, um, use MCP servers as well, and I think earlier in the year I made some really fun demos being able to just really quickly combine tools for a prototype. **Host** [15:40] Yeah. **Nina Lopatina** [15:40] So I think it's really helped people prototype faster and show value in an early version to then kind of build out in larger scale. And I think for us, uh, for me personally, like in my dynamic agent configs, I'm moving more toward API calls and something a little bit more, uh, once I've kind of maybe been able to prototype with an MCP server and figure out how I'm gonna use this, uh, I think then you can cause... you re- can reduce the complexity- **Host** [16:05] Yeah **Nina Lopatina** [16:06] ... and reduce that dependency. **Host** [16:07] Yeah. Um, mentioning MCP, I don't know how far you wanna go into this, but, um, the MCP gateway finally launched from Anthropic. There's a bunch of MCP sort of services that are doing various sort of discovery and being a registry. What should people know? What, what, what are people betting on? What's, what's, what's actually working? **Nina Lopatina** [16:27] In terms of MCP servers or directories? **Host** [16:29] Uh, yeah, directories, gateway, auth, anything of that nature. Basically, like everything that happens after the initial launch of MCP. **Nina Lopatina** [16:37] Yeah. **Host** [16:38] Like there's been a bunch of work. There's MCP UI, but I don't know if that's really strictly context engineering. So... **Nina Lopatina** [16:44] Yeah. We added our MCP server to the registry, and it really seemed like, uh- **Host** [16:50] The official Anthropic one. **Nina Lopatina** [16:51] Yeah. **Host** [16:51] Okay. **Nina Lopatina** [16:52] Yeah, and it really seemed like that registry is meant to be read by agents, not humans. **Host** [16:56] Yeah. **Nina Lopatina** [16:56] Yeah. **Host** [16:57] This is like a GitHub repo. **Nina Lopatina** [16:58] Yeah, yeah. Which is interesting because I think, yes, there's definitely value to having strong agent experience, and I think that's gonna be an area that's gonna be growing, uh, over the next year for sure, is being able to let agents just, like, do things without a human in the loop. But I do think for MCP servers, I think you kind of still want a human to make that selection of what you're going to include in your list of tools. You wanna check, you know, the security and, uh, other, other things before you would let it run. **Host** [17:26] Yeah. Amazing. Okay. So more broadly, I guess any other things you call out in like the state of engineering, any, uh, so context engineering, any other good work by other companies? You ca- shouted out Manus briefly. ### Optimizing Context **Nina Lopatina** [17:37] Yeah. I think there's some been, there's been some really interesting research, let's say, in, uh, optimizing the system prompt. **Host** [17:44] Okay. **Nina Lopatina** [17:45] So- **Host** [17:45] This is the, uh, continual prompt learning from Arize or- **Nina Lopatina** [17:49] No, I'm thinking- **Host** [17:49] JePA **Nina Lopatina** [17:50] ... JePA. **Host** [17:50] Um, I think people are very excited about JePA. The way that I explain it, you, you can feel free to correct me, is it's kind of like an evolution of the original DS5 idea where you set objectives and let LLMs optimize their own prompts by looking at output and thinking about and reasoning- About, like, what, what should they continue to add in the prompts to improve their evals. Um, so it's like a nice, like, sort of PyTorch-like model of a, a training loop, but it's only in the prompts, not in the weights. **Nina Lopatina** [18:21] Mm-hmm. **Host** [18:21] You can also obviously extend it to the weights. And I think the other thing about why it's called JePA, there's an evolutionary element or genetic evolutionary element where you roll out multiple samples and you select the, the best- **Nina Lopatina** [18:33] Mm-hmm **Host** [18:33] ... survivors from that. Anything else I missed? **Nina Lopatina** [18:36] Uh, no. That captures it really well and actually jogs my memory of ACE. **Host** [18:40] Yeah. **Nina Lopatina** [18:41] So, uh, so actually agentic context engineering, that approach actually has shown better benchmark performance on financial and other complex document sets, and the approach they've taken is quite interesting. So basically, if you take an approach like JePA and you basically, like, maybe throw out the whole prompt and start over, or you do, like, many, many steps and you kind of, like, compress and expand and compress and expand, you're kind of gonna lose some information and you can see a significant drop-off in performance. And so they're using this agentic approach to just make smaller tweaks in the, in the current prompt rather than rewriting from scratch. **Host** [19:16] Yeah. **Nina Lopatina** [19:16] That's among their innovations for that approach that I think is actually kind of goes along with what Teh said about the KV cache and really, like... I think, um, agents can get pretty confused pretty quickly, and that can really degrade performance and cause hallucinations. And so I think wherever you can, like, one, use the KV cache for both efficiency, but also for some sort of, like, more stable environment for the agent to be able to take, um, you know, more and more actions in. **Host** [19:42] Yeah. How much KV cache decision-making is there for context engineering? I feel like obviously, obviously the answer is the stuff that doesn't change, put it up front, and the stuff that does change a lot, put it at the bottom, right? Uh, I don't care, and I think that's mostly because most agents that I care about are multi-turn, and so the cache is the whole turn, the, the, the, the, the five turns that happened before, and I'm not changing the system prompt that much, right? So cache, KV cache would... You save money if you have... You're serving the same prompt to 1,000 customers. Like, I guess, I guess that's it, but we don't do that, so I don't know. **Nina Lopatina** [20:20] Yeah. I think it can also improve performance, but I think as conversations take more and more turns, I haven't really seen a system that handles this well. Like, for Cursor, I get to a certain point in the conversation and I'm just, I'm opening a new window even if I wasn't done with that conversation- **Host** [20:36] Mm. Yeah **Nina Lopatina** [20:36] ... because the context floats and you- **Host** [20:39] Yeah. So, so, um, you know, Dex would call this, like, intentional context compression, intentional frequent context compression, 'cause, like, you don't trust the model to do co- co- compaction just yet. **Nina Lopatina** [20:48] Mm-hmm. **Host** [20:48] I would say that both Anthropic and I think OpenAI and maybe Gemini as well, they're all doing, like, compaction inside the model with the, the, uh, each, literally each of their frontier releases. I don't know if that's interesting to you, to you or, yeah. **Nina Lopatina** [21:02] Yeah. **Host** [21:03] Any evals on it? ### Embodied Evals **Nina Lopatina** [21:03] Well, actually, I did a, I like, what I like to call an embodied eval of ChatGPT last winter. I used it as a training coach for a snowboarding mogul race. **Host** [21:16] What is a mogul race? You, you snowboard around moguls? **Nina Lopatina** [21:18] Yeah. **Host** [21:19] Okay. **Nina Lopatina** [21:19] Like, a double diamond mogul run and you lap 25 times. And I- **Host** [21:23] 25? That's a lot. **Nina Lopatina** [21:25] Yeah, it's about 40,000 vertical feet. It's usually 23. **Host** [21:28] All right. **Nina Lopatina** [21:28] So, right, I mean, training is very important to be able to do this and to do it safely. **Host** [21:33] It's also a workout. I don't know how calories you burn on that. **Nina Lopatina** [21:35] A lot. They give you a big pasta dinner at the end. Um, and so this was a really long-term project. It was, like, three or four months, and also it required me taking action in the real world in an embodied way and then, uh, having that loop. And I just had to, like, um, close that window and, like, restart, and then I lost a lot of training info. So I think, like, that's kind of made me, I think, pre- proactively, uh, limit the number of turns, so maybe I'm missing some of the progress that's happened since then. This season I'm evaluating multiple models to see who is the best coach. **Host** [22:09] Yeah, but then you have to, like, copy-paste, do data entry on all these models, right? It's... That's not great. Unless you have your own custom interface or are you just going to chatgpt.com- **Nina Lopatina** [22:19] Yeah **Host** [22:19] ... claude.ai? **Nina Lopatina** [22:20] I think I'm gonna do, like, initial interviews on LM Arena. **Host** [22:22] Okay. I see, I see. **Nina Lopatina** [22:23] And then kind of- **Host** [22:24] Yeah, yeah **Nina Lopatina** [22:24] ... maybe pick one or two. **Host** [22:25] Yeah, yeah. **Nina Lopatina** [22:26] Yeah. **Host** [22:26] Interesting. Okay, cool. Um, automated context engineering is really great. Any specifically for code, I guess? I, I don't know how, how much you guys encounter code. Uh, it sounds like, sounds like yes. Um, and is it very different than, like, legal and retail and support, all these other domains? **Nina Lopatina** [22:42] Yeah. So our goal has been to create a platform that's really end to end and easy to get up and running for any domain and any use case. And, you know, I think we saw that firsthand, uh, with kind of a, you know, a recent beta at that hackathon I mentioned for e-commerce, although we do have customers in that domain, but I haven't, uh, really interacted with that work very much. And so code has been one of the domains we've worked with as well. And so we actually just used our platform for, uh, test code generation for devices. And what we found is kind of the same approach we take where we have, uh, multimodal ingestion and an ability to get the hierarchy of the document contents and then retrieval pipeline that includes, you know, filters, re-rankers, hybrid search. All of that combined is a really great starting point, and then we're able to hill climb very quickly and, uh, actually had state-of-the-art or I guess, uh, the highest, uh, human-based evals for that customer compared to- **Host** [23:47] Yeah **Nina Lopatina** [23:48] ... you know, coding platforms. So I think it's really... I mean, it requires a bit of customization, but I think context engineering, you know, applies to code the same way that it applies to other domains. **Host** [23:59] Yeah. Awesome. A little bit of prediction corner here. What do you think is underrated now in context engineering that will be a big con- topic conversation next year? ### Looking Ahead **Nina Lopatina** [24:08] Hmm. **Host** [24:09] Like, what, basically what's underrated, right? Like, what, what, what are, what should people talk about more but they're not? **Nina Lopatina** [24:16] I think really the full system. So I think right now what people are talking about are innovations in one part of the system or another, like let's say different components like a memory system or a re-ranker or, you know, design patterns around compressing context, things like that. And I think in the next year we'll have full systems that can be kind of a design pattern rather than... And, and ha- being a-- having a discussion at that level rather than- **Host** [24:45] Yeah **Nina Lopatina** [24:45] ... components. **Host** [24:46] Yeah. Amazing. This is your fifth NeurIPS. I would say like, well, for people who've been long-timers, I think the last time that NeurIPS was in San Diego was the Florida NeurIPS. I don't know if you, you were there for that. **Nina Lopatina** [24:58] Which year was that? **Host** [24:59] Like 2017 or something. Yeah. Um, how did you reflect on the scene changing, uh, when you come back? **Nina Lopatina** [25:07] Yeah, it's so interesting. So my first NeurIPS was in 2016 in Barcelona. **Host** [25:12] Ah. **Nina Lopatina** [25:12] So- **Host** [25:13] That's nice. **Nina Lopatina** [25:13] Yeah. Uh, so I was, uh, actually, I had just finished my graduate research in neuroscience in reward learning and decision-making. I was doing a postdoc at Berkeley, but I had a poster from my graduate work, so I flew out for a few days, and it was, uh, really not what I expected from a research conference, having mostly attended neuroscience research conferences. **Host** [25:34] Oh, yeah. How's it different? **Nina Lopatina** [25:35] Uh, so- **Host** [25:35] This is neural information processing. **Nina Lopatina** [25:37] I know. I know. So I've run into so many neuroscientists here as well, and people I know- **Host** [25:42] Yeah **Nina Lopatina** [25:42] ... from that, um, earlier career stage. Uh, I was not expecting all the industry parties. That's just not a thing in neuroscience. **Host** [25:49] Yeah. **Nina Lopatina** [25:49] Yeah. **Host** [25:50] Yeah. **Nina Lopatina** [25:50] And it was just such a smaller... You know, it seemed like a large conference, and everyone there was like, "Oh, yeah, it used to be much, much-" **Host** [25:59] Quieter. Smaller. **Nina Lopatina** [26:00] Yeah, smaller. And I think now I'm just like, you know, there's so many people, it's their first year, and I'm just like not seeing some of the same folks I ran into in the earlier years. **Host** [26:10] They're still around. You just have to find them in parties. And like spaces like LAO Lounge. Very cool. Uh, any call to action? How can people help you, find you, anything like that? **Nina Lopatina** [26:20] Yeah. We have some really exciting updates coming in the domain of context engineering. So, you know, follow me, Nina Lopatina, on Twitter or LinkedIn or Contextual AI, and, uh, just, uh- **Host** [26:32] Stay tuned. **Nina Lopatina** [26:33] Yeah. Stay tuned. **Host** [26:34] Awesome. Thank you. **Nina Lopatina** [26:35] Thanks. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.