# Long Live Context Engineering - with Jeff Huber of Chroma Latent Space · 2025-08-19 Jeff Huber, founder and CEO of Chroma, joins Shawn Wang and Alessio to argue that context engineering—not RAG—is the real job of AI builders, and that Chroma's modern retrieval engine is built for this new paradigm. Huber explains context rot, a phenomenon where LLMs lose attention and reasoning ability as context windows grow, motivating precise retrieval over huge context windows. Chroma Cloud is a zero-config, usage-based serverless vector database with separation of storage and compute, now powering 5M monthly downloads and 21k GitHub stars. He shares techniques like chunk rewriting and generative benchmarking for evaluating retrieval, and predicts LLMs will increasingly serve as re-rankers. The episode also covers memory as a benefit of context engineering, offline compaction for self-improving AI systems, and Huber's philosophy of building with long-term purpose, hiring engineers who care about Rust, TLA+, and deterministic simulation testing. ## Questions this episode answers ### What is 'context engineering' as defined by Jeff Huber of Chroma? Jeff Huber defines context engineering as the job of figuring out what should be in the context window for any given LLM generation step. He describes an inner loop (choosing information for the current step) and an outer loop (improving selection over time). He argues it's a high-status job and what most successful AI startups are fundamentally good at, elevating the role beyond vague terms like RAG. [12:15](https://addtry.com/20e44b6d-2eaf-47e9-9a3c-18522b5ea93a?t=735000) ### What is 'context rot' and how did Chroma's technical report compare models like Claude and GPT-4.1 on handling large context windows? Context rot is the finding that LLM performance is not invariant to token count — as more tokens are added, the model pays less attention and reasons less effectively. Chroma's report showed Claude Sonnet-4 had the best area under the curve for maintaining performance, while Qwen, GPT-4.1, and Gemini Flash degraded more quickly. Jeff Huber noted that needle-in-a-haystack benchmarks don't capture real-world behavior. [16:12](https://addtry.com/20e44b6d-2eaf-47e9-9a3c-18522b5ea93a?t=972000) ### How does Chroma's CEO describe 'modern search for AI' and what makes it different from traditional search? Jeff Huber says modern search for AI differs in four ways: the tools and technology (like Rust-based distributed systems with separation of storage and compute), the workload (LLMs can digest orders of magnitude more results than the ten blue links humans consume), the developer (AI app developers have distinct needs), and the result consumer (now a language model, not a human doing the last‑mile analysis). These factors shape how retrieval systems are designed. [3:10](https://addtry.com/20e44b6d-2eaf-47e9-9a3c-18522b5ea93a?t=190000) ## Key moments - **[0:00] Intro** - **[0:48] Why Chroma** - [0:56] The gap between AI demo and production felt more like alchemy than engineering, says Jeff Huber. - [2:17] Jeff Huber: "Doing a database is not a side quest, it is part of the main quest." - **[2:55] Modern Search** - [3:04] Q: Are information retrieval and search the same thing? Jeff Huber says modern search for AI differs in tools, workload, developer, and consumer. - **[4:29] Staying Focused** - [5:16] Jeff Huber: "If you follow lean startup methodology, you'll probably end up building a dating app for middle schoolers." - [7:22] Jeff Huber: "The upstream version of Conway's Law is you ship your culture." - **[8:08] Chroma Cloud** - [8:09] Chroma reaches 5 million monthly downloads and 21,000 GitHub stars, with over 60 million all-time downloads. - [11:36] Users can sign up on Chroma Cloud, create a database, and load data in under 30 seconds. - **[12:15] Context Engineering** - [12:36] Context engineering is the job of figuring out what should be in the LLM context window, with an inner and outer loop. - [14:14] Chroma's context rot technical report shows LLMs pay less attention and reason less effectively as token count increases. - [17:43] Claude Sonnet-4 shows best area under curve on context rot benchmark, followed by Qwen, GPT-4.1, and Gemini Flash. - [18:27] Jeff Huber: "I would rather have a model with a 60,000 token perfect window than a 5 million token window." - [21:03] Shawn Wang vs Jeff Huber: Do AI labs care about context engineering for consumers? Shawn says yes due to ChatGPT memory, Jeff says they won't share secrets. - **[21:49] Retrieval Strategies** - [23:25] Jeff Huber: "A lot of people are just still yeeting everything into the context window." - [24:37] A dominant pattern emerges: first-stage retrieval narrows to 300 chunks, then an LLM re-ranker selects the top 30. - [25:34] Jeff Huber predicts purpose-built re-ranker models will disappear as LLMs become cheaper and faster. - **[27:02] Code Indexing** - [27:02] Q: How should code indexing be approached? Jeff Huber says indexing trades write time for query time, and Regex is powerful for code search. - [29:13] Chroma's forking feature creates a copy of an index in under 100 ms for pennies, enabling per-commit search. - [30:24] Jeff Huber: Full text search excels when the query writer knows the data; semantic search when they don't. - [33:13] Jeff Huber urges developers to create small golden datasets of query-chunk pairs to quantitatively evaluate retrieval. - **[34:07] Future Retrieval** - [34:34] Jeff Huber: "Why are we going back to natural language? Why aren't we passing embeddings directly?" - [36:03] Swyx: "Agentic RAG." Jeff Huber: "Stop. Don't call it that." - **[38:06] Memory** - [38:07] Jeff Huber: "Memory is the benefit of context engineering." He says memory makes AI usable like training a human. - [42:34] Chroma's Generative Benchmarking technical report shows how to teach an LLM to generate query-chunk pairs for evaluation. - **[45:46] Lessons & Culture** - [45:46] Jeff Huber's lesson from past startups: only do work you love with people you love; don't make trade-offs. - [48:46] Jeff Huber: "AGI is also a religion. It has a problem of evil" and promises salvation through infinite intelligence. - [50:00] Q: Who designs Chroma's cohesive branding? Jeff Huber credits talented designers and emphasizes intentional, consistent experience. - [52:43] Chroma is hiring product designers, researchers, and distributed systems engineers who care about Rust, TLA+, deterministic simulation. - [55:51] Jeff Huber reports that AI coding tools like Cursor and Windsurf are still not great at Rust due to limited training data. ## Speakers - **Alessio** (host) - **Swyx** (host) - **Jeff Huber** (guest) ## Topics Data Tools, Benchmarks ## Mentioned Anthropic (company), Chroma (company), Know It (company), Mighty Hive (company), OpenAI (company), Pinecone (company), Standard Cyborg (company), Weaviate (company), ChatGPT (product), Chroma Cloud (product), Chroma Distributed (product), Claude (product), DeepSeek-R1 (product), GPT-4.1 (product), Gemini Flash (product), LangChain (product), LlamaIndex (product), Qwen (product), SQLite (product), VS Code (product) ## Transcript ### Intro **Alessio** [0:05] Hey everyone, welcome to the Latent Space Podcast in the new studio. This is Alessio, partner and CTO of Decibel, and I'm joined by Swyx, founder of Small AI. **Swyx** [0:12] Hey, hey, hey. It's weird to say welcome because obviously, actually, today's guest, Jeff, has welcomed us to Chroma for many months now. Welcome. **Jeff Huber** [0:21] Thanks for having me. Good to be here. **Swyx** [0:23] Jeff, you're a founder, CEO of Chroma. I've sort of observed Chroma for a long, long time, especially way back in the, the old office. And, uh, you were-- You originally sort of got your start in the open source vector database, right? **Jeff Huber** [0:35] Yeah. **Swyx** [0:35] Like you sort of, you're the open source vector database of choice of, of a lot of different projects, uh, particularly with even, even projects like the, uh, Voyager paper, you guys were used in that. I don't even know, like, the, the, the full list. **Jeff Huber** [0:46] Mm-hmm. **Swyx** [0:47] But, um, how do you introduce Chroma today? ### Why Chroma **Jeff Huber** [0:49] It's a good question. I mean, naturally, you always wanna kind of take your messaging and make it fit your audience. **Swyx** [0:55] Yeah. **Jeff Huber** [0:56] But I think the reason that Chroma got started is because we had worked for many years in applied machine learning, and we'd seen how demos... Demos were easy to build, but building a production reliable system was incredibly challenging, and that the gap between demo and production didn't really feel like engineering. It felt a lot more like alchemy. There's some good, like, XK-XKCD memes about this guy standing on top of a giant steaming pile of garbage, and, uh, the other character asks, "This is your data system?" And he's like, "Yes." And he's like, "How do you know if it's good, or how do you make it better?" "Oh, you just, like, stir the pot and then, like, see if it gets any better." That just seemed intrinsically wrong, and this is back in, like, twenty twenty-one, twenty twenty-two that, like, we were having these conversations. And so that coupled with, like, a thesis that, like, latent space was a very important tool. That, that is a plug, yes. **Swyx** [1:44] We agree. **Jeff Huber** [1:45] That is a plug. **Swyx** [1:46] We need to- **Jeff Huber** [1:47] Yeah **Swyx** [1:47] ... ring the bell. **Jeff Huber** [1:48] Yeah, exactly. Ring the gong. That latent space, both the podcast, but also the technology, was a very underrated tool and a very, like, important tool for interpretability. It's fundamentally how models see their own data. We as humans can kind of, you know, have that shared space to understand what's going on. That's where we got started, and so I think that's also where we continue to want to go. Like, what do we want to do? We want to help developers build production applications with AI and want to make the process of going from demo to production feel more like engineering and less like alchemy. Doing a database is, like, not a side quest. It is a part of the main quest. What we realized along the way was search was really a key workload to how, like, AI applications were gonna get built. It's not the only workload, but it's, like, definitely a really important workload. And that you don't earn the right to do more things until you've done one thing at a world-class level. That requires maniacal and, you know, kind of, uh, maniacal focus. Um, and so that's really what we've been doing for the last few years. That was a long, kind of rambling introduction, but, like, maybe to sort of land the plane, you know, if you ask people, you know, "What does Chroma do today?" We build a retrieval engine for AI applications. We're working on modern search infrastructure for AI, um, some version of that. ### Modern Search **Swyx** [2:56] I'll do a double click on this. Is information retrieval and search the same thing, or are they slightly different in your mind? I just wanted to clarify on terminology. **Jeff Huber** [3:04] Yeah, I think that, you know, that modern search infrastructure for AI- **Swyx** [3:08] Yeah **Jeff Huber** [3:08] ... we can maybe unpack that for a couple seconds. **Swyx** [3:10] Yeah, yeah. **Jeff Huber** [3:10] So modern is in contrast to traditional, and mostly what that means is, like, modern distributed systems. So there's a bunch of primitives in building great distributed systems that have come on to the scene in the last five, ten years that obviously are not in technology that is older than that by definition. Separation of read and write, separation of storage and compute. Chroma's written in Rust, deployable to tenant. Um, we have-- We use object storage as a key persistence tier and, like, data layer for Chroma, uh, distributed in Chroma Cloud as well. So that's the modern piece. And then the for AI piece actually, I think is-- it matters in four kind of different ways. Like, for AI means four different things. Like, it means, number one, the tools and technology that you use for search are different than in classic search systems. Number two, the workload is different than classic search systems. Number three, the developer is different than classic search systems. **Swyx** [3:59] Hmm. **Jeff Huber** [4:00] And number four, the people whose-- the person who's consuming those search results is also different than in classic search systems. Think about, like, classic search systems, like, you as the human were doing the last mile of search. You know, you were doing- **Swyx** [4:12] Click, click, click, click. **Jeff Huber** [4:12] Exactly. You're like, "Oh," like- **Swyx** [4:14] New tab **Jeff Huber** [4:14] ... which of these are relevant, open a new tab, summarize, blah, blah, blah, blah. You the human were doing that, and now it's a language model. Humans can only digest ten blue links. Language models can digest orders of magnitude more. All of these things matter and I think influence, like, how a system is designed and what it's sort of, like, made for. ### Staying Focused **Swyx** [4:30] Back in twenty twenty-three, I think the vector DB category was kinda one of the hottest ones, and you had Pinecone raise a hundred million. You had all these different, Weaviate. **Jeff Huber** [4:38] Yeah. **Swyx** [4:38] You had all these companies. **Jeff Huber** [4:39] Yeah. **Swyx** [4:40] How did you stay focused on, like, what mattered to you, rather than just try to raise a lot of money and make a big splash? And it took you a while to release Chroma Cloud too- **Jeff Huber** [4:49] Mm **Swyx** [4:49] ... which rather than just getting something out that maybe broke once you got to production, you kinda took your time. **Jeff Huber** [4:54] Yeah. **Swyx** [4:54] Can you maybe give people advice on, in the AI space, how to be patient as a founder and how to have your own vision that you follow versus kinda like following the noise around you? **Jeff Huber** [5:04] There are different ways to build a startup, and so, you know, there are different schools of thought here. So one school of thought certainly is, like, the find signal and kinda follow the gradient descent of what people want, sort of lean startup style. My critique of that would be that if you follow that methodology, you'll probably end up building a dating app for middle schoolers, because that just seems to be, like, the lowest base take of what humans want to some degree. The slot machine would be the AI equivalent of that. Versus, you know, the other way to build a startup is to have a very strong view, presumably a contrarian view, or at least a view that seems like a secret, and then to just be maniacally focused on that thing. You know, there are different strokes for different folks, but we've always taken the second approach. And yeah, there was the option of like, okay, Chroma's single node is, like, doing really well, getting a bunch of traffic. Clearly, having a hosted service is the thing people want. Like, we could just spend, uh, we could very quickly get a product in the market. But we felt like, no, really what we want Chroma to be known for is our ... developer experience. Like, we want our brand to be... We want Chroma's brand and the craft expressed in our brand to be extremely well known. And we felt like by offering a single node product as a surface, like, it was not gonna meet our bar of, like, what great developer experience could and should look like. Yeah, we made the decision of like, no, we're gonna, like, build the thing that we think is right, which was really challenging, um, and took a long time. And obviously we're, we're incredibly proud that it exists today and that it's, like, serving hundreds of thousands of developers and they love it, but it was hard to get there. **Alessio** [6:39] When you're building the team, how do you message that? If I go back maybe like a year and a half ago, you know, I could join Chroma, I could join all these- **Jeff Huber** [6:46] Yeah **Alessio** [6:46] ... different companies. How do you keep the vision clear to people when on the outside you have, oh, I'll just use PG Vector or like, you know, whatever else the thing of the day is. **Jeff Huber** [6:56] Mm. **Alessio** [6:57] Do you feel like that helps you bring people that are more aligned with the vision versus more of the missionary type on just joining this company before it's hot and maybe, yeah, any learning that you have from recruiting early on? **Jeff Huber** [7:08] The upstream version of Conway's Law, like you ship your org chart, is you ship your culture, 'cause I think your org chart is downstream of your company's culture. We've always placed an extremely high premium on that, on the people that we actually have here on the team. Um, I think that the slope of our future growth is entirely dependent on the people that are here in this office. And, you know, that could mean going back to zero, that could mean, you know, linear growth, that could mean all kinds of versions of, like, hyper linear growth, exponential growth, hockey stick growth. And so yeah, we've just really decided to hire very slowly and be really picky. And I don't know. I mean, you know, the future will determine whether or not that was the right decision, but I think having worked on a few startups before, like, that was something that I really cared about, was, like, I just wanna work with people that I love working with and, like, want to be shoulder to shoulder with in the trenches, and I think can independently execute on the level of, like, craft and quality that, like, we owe developers. And so that was how we chose to do it. ### Chroma Cloud **Swyx** [8:09] Talk about standard condition on, on the other fun stuff towards the end, but we'll, we'll focus on Chroma. **Jeff Huber** [8:13] Great. **Swyx** [8:13] I, I always wanna put, like, some headline numbers up front. So I'm just trying to do a better job of, like, giving people the brain dump on what they should know about Chroma. **Jeff Huber** [8:21] Yeah. **Swyx** [8:21] 5 million monthly downloads is, is what I have on PyPi. **Jeff Huber** [8:24] Mm-hmm. **Swyx** [8:25] And 21,000 GitHub stars. **Jeff Huber** [8:27] Mm-hmm. **Swyx** [8:27] Anything else people should know, like, that's like the typical sales call, like, headline stuff like that, you know? **Jeff Huber** [8:33] Yeah. Um, yeah, 20,000 GitHub stars, 5 million plus monthly downloads. Um, I haven't looked at the number recently. I think it's, like, over 60 or 70 million all-time downloads now. **Swyx** [8:42] Yeah. **Jeff Huber** [8:43] For m- many years running, Chroma's been the number one used- **Swyx** [8:45] Open source **Jeff Huber** [8:45] ... project. **Swyx** [8:46] Yeah. **Jeff Huber** [8:46] Broadly, but also within communities like LangChain and LlamaIndex. **Swyx** [8:50] Okay. Cool. Fair enough. Yeah, yeah, I think, like, when you say single node Chroma, like, I think you're describing the core difference between, like, what Chroma Cloud has been, and I think we're- **Jeff Huber** [9:00] Yeah **Swyx** [9:00] ... releasing this in, in line with, like, your GA in Chroma Cloud. Um- **Jeff Huber** [9:04] Yes **Swyx** [9:05] ... so, like, what should people know about Chroma Cloud and, like, how you've developed this experience from, from the start? Like, you, you, you mentioned separation of storage and compute. Like, what does that entail? **Jeff Huber** [9:13] Yeah, 100%. Chroma's known for its developer experience. I don't know that we were the first to do this, but I think we were. With Chroma, you just pip install chromadb, and then you can use it. **Swyx** [9:23] It's just, like, in memory. **Jeff Huber** [9:24] Like, I think it may have been the first- **Swyx** [9:25] You, you can persist it in the database **Jeff Huber** [9:26] ... it could've been the first database to ever be pip installable. Um, but- **Swyx** [9:29] Any SQLite wrapper is pip installable technically, you know. **Jeff Huber** [9:33] No, SQLite was not, like, pip installable even to this day, I don't think. **Swyx** [9:36] You probably have a, a, a deeper dive and knowledge of this. I, I'm just speculating myself. **Jeff Huber** [9:40] Yeah. So that, that led to, like, a very seamless onboarding experience for new users, 'cause you could just run a command, and then you could use it. We did all the work to make sure that, like, regardless of the deployment target or architecture that you're running it on, like, it would just work. In the early days, we had people do really weird stuff, like run it on Arduinos and PowerPC architectures and, like, really esoteric stuff. Like, we would, like, go the extra mile to, like, make sure that it worked everywhere and just, it just always worked. So that was Chroma single node. So going back to, like, the developer experience we wanted to have in a cloud product, like, we thought that in the same way that you could run pip install chromadb and be up and running in five seconds and, like, not have to think about it, you have to learn a bunch of, uh, abstractions. You don't have to, like, spend a bunch of time learning this really complicated API. That same story had to be true for the cloud. And so what that meant is, like, having a version of the product where you have to be forced to think about, like, how many nodes you want or how to size those nodes, or how your sharding strategy should be, or your backup strategy, or your data tiering strategy, or I could go on. Like, that just wasn't, wasn't good enough. It needed to be, like, zero config, zero knobs to tune. It should just be always fast, always very cost effective, and always fresh without you having to do or think about anything, regardless of how your traffic goes up and down and how your data scale goes up and down. That was sort of the, the motivating criteria. It also, like, usage-based billing model was really important because that just is, like, so fair. We only charge you for the minimal slice of compute that you use and, like, nothing more, which not all serverless databases can claim, but it is true inside of Chroma that we, like, truly only charge you for the narrow slice of what you use. And so, like, that was the criteria that we entered kind of the design criteria process in. Um- **Swyx** [11:20] Which is, you know, de facto, you're also building a serverless compute platform. **Jeff Huber** [11:24] Yeah, you have to. No, y- exactly. That motivated the design of Chroma Distributed. Chroma Distributed is also a part of the same monorepo that's open source Apache 2. And then the control and data plane are both fully open source Apache 2. And then Chroma Cloud uses Chroma Distributed to run a service. And that service you can sign up, create a database, and load in data in under 30 seconds. And at least as, sort of at time of filming, people get, like, five bucks of free credits, which is actually enough to load in, like, 100,000 documents and query it 100,000 times, which obviously for a lot of use cases actually might mean they use it for free for years, which is fine. And to get there, we had to do kind of all the, all the hard work. **Swyx** [12:03] Yeah. I think every blog should basically have semantic indexing. So, like, you know, you host your personal blog on Chroma. You know, like, why not? **Jeff Huber** [12:11] Yeah, I mean, you know, the, the mission of organizing the world's information remains unsolved. Yeah. ### Context Engineering **Swyx** [12:16] You have one of your usual cryptic tweets, a new text, uh, you tweeted context engineering a couple months ago. What was it? **Jeff Huber** [12:23] Mm. **Swyx** [12:23] Uh, April. I, I think everybody now is talking about context engineering. Can you give the canonical definition for you, and then how, um, Chroma plays into it, and then we'll talk about all the different pieces of it. **Jeff Huber** [12:36] I think something that's incredibly important when a new market is emerging is abstractions and the primitives that you use to reason about that thing. And AI, I think like in part of its hype, has also had a lot of primitives and abstractions that have gotten thrown around and have led to a lot of developers not actually being able to think critically about what is this thing, how do I put it together, what problems can I solve, what matters, where should I spend my time? For example, the term RAG. We never use the term RAG. Like, I hate the term RAG. **Swyx** [13:08] Yeah, I killed the RAG track, uh, partially because of your influence. **Jeff Huber** [13:11] Thank you. Thank you. A, A is just retrieval, first of all. Like retrieval, augmented generation are three concepts put together into one thing. Like, that's just really confusing. And of course, RAG got known now as it's branded as like, you know, oh, you're just using single dense vector search, and that's what RAG is. It's also dumb. I think one of the reasons I was really excited about the term, I mean, obviously, AI engineering, which you did a ton of work for, like context engineering is in some ways a subset of AI engineering. Like what is it? It's a, it's a high status job. Context engineering is the job of figuring out what should be in the context window, any given LLM generation step. And there's both an inner loop, which is setting up the in fish-- you know, what should be in the context window this time, and there's the outer loop, which is how do you get better over time at filling the context window with only the relevant information. And we recently released this technical report about context rot, which goes sort of in detail and depth about how the performance of LLMs is not invariant to how many tokens you use. As you use more and more tokens, the model can pay attention to less, and then also can reason sort of less effectively. I think this really motivates the problem. You know, context rot implies the need for context engineering. And I guess like why I'm really excited about the meme, and, you know, I got, I got maybe both lucky, uh, to some degree that, you know, called it back in April, this is gonna be a big meme, is that it elevates the job to... It, it clearly def-def describes the job, and it elevates the status of the job. This is what frankly, most AI startups, any AI startup that you know of that you think of today that's doing very well, like what are they fundamentally good at? What is the one thing that they are good at? It is context engineering. **Swyx** [14:46] Particularly, I would feel like the, the, the p- a lot of pieces I've read, a lot of it focuses on agents versus non-agent stuff, like the context engineering is more relevant for agents. Do you make that distinction at all, or you're just looking at context engineering generally? **Jeff Huber** [15:00] No. I think there... I mean, there's interesting agent implications of like, you know- **Swyx** [15:04] Yeah **Jeff Huber** [15:04] ... agent learning. Can, you know, can agents kind of learn from their interactions, which maybe are less relevant than like static sort of knowledge-based corpuses, chat your documents, obviously. Then again, like, you know, I think you could make the argument that even like chat your document use cases like should get better with more interactions. I don't draw a distinction between agent and non-agent. I don't actually know what agent means still, but again, primitives, abstractions, words, they matter. I don't know. Like I- **Swyx** [15:29] Yeah **Jeff Huber** [15:29] ... what does agent mean? I don't know. **Swyx** [15:30] Well, there's many definitions out there. **Jeff Huber** [15:32] Exactly. **Swyx** [15:32] I've taken a stab. **Jeff Huber** [15:33] Most terms that can mean anything are just a vehicle for people's hopes and fears. **Swyx** [15:39] Yeah. Mm. **Jeff Huber** [15:40] Um, I think, you know, agent is the same- **Swyx** [15:42] Yeah **Jeff Huber** [15:42] ... thing. **Swyx** [15:42] For sure. Well, maybe we'll try to be more concise or precise about context engineering so that it doesn't, uh, it actually means something, and- **Jeff Huber** [15:48] Yeah. Great **Swyx** [15:48] ... you know, people can actually use it to, to do stuff. Uh, one thing I definitely will call out for context engineering or context rot in general is I think that there's been a lot of marketing around needle in a haystack- **Jeff Huber** [15:58] Mm **Swyx** [15:58] ... where every frontier model now comes out with like completely green perfect charts of like full utilization across, you know, 1 million tokens. And I'm wondering what you guys' , uh, takes are on, on, on that kind of marketing and- **Jeff Huber** [16:10] Yeah. **Swyx** [16:11] Yeah. **Jeff Huber** [16:12] So maybe to back up a little bit, the way that we came to work on this research was we were looking actually at agent learning. So we were just very curious, like could you give agents access to like prior successes or prior failures? And if you did, would that help boost agent performance? So we're specifically looking at a couple different data sets, uh, Sweet Bench inclusive, and we started seeing interesting patterns where like on sort of multi-turn agent interactions, we were giving it the whole conversation window, like the number of tokens explodes extremely quickly. And instructions that were clearly in there, like were being ignored and were not being enacted upon. And we're like, oh, that, that clearly is a problem. We've now felt the pain. It was sort of a meme amongst people in the know that like this was true, and like I think also, you know, some of the research community's reaction to the context rot technical report is like, yeah, we know. And you know, that's fine. Nobody else knew, and like it'd be kind of nice if like you can actually teach builders what is possible today versus what is not possible today. **Swyx** [17:03] Yeah. **Jeff Huber** [17:03] I don't blame the labs. I mean, building models is so insanely competitive. Everybody invariably is like picking the benchmarks that they want to do the best on. They're training around those. Those are also the ones that are, you know, find their way into their marketing. You know, most people are not motivated to come out and say, "Here are all the ways that the, that our thing is great, and here are the ways that our thing is not great." You know, I don't know. I can have, I have some sympathy for, you know, why this was not reported on. But yeah, I mean, there was, there was this bit of like this sort of implication where like, "Oh look, our model is perfect on this task, needle in a haystack, therefore, the context window you can use for whatever you want." There was an implication there, and while I hope that that is true someday, that is not the case today. **Swyx** [17:43] Yeah. We'll, uh, send people w- at least on the YouTube video, we'll put this chart, which is kind of your figure one of the context rot report. It seems like Cl- Sonnet-4 is the best in terms of area under curve is how to think about it. Then Qwen, wow, and then GPT-4.1 and Gemini Flash are, are, uh, degrade a lot quicker- **Jeff Huber** [18:02] Mm **Swyx** [18:02] ... in terms of the context length. **Jeff Huber** [18:03] Yep. I don't have much commentary. That is what we found- ... uh, for this particular task. **Swyx** [18:08] Yeah, yeah. **Jeff Huber** [18:08] Again, how that translates to people's actual experience in real world, you know, tasks is entirely different. I mean, there is a certain amount of love that developers have for Claude, and like maybe those two things are correlated. **Swyx** [18:18] Yeah, I think it shows here if, if this is, this is true, that's, that's a big explanation for why **Jeff Huber** [18:23] You follow my instructions, you know? Like, here's a clear baseline, uh- **Swyx** [18:27] Yeah **Jeff Huber** [18:27] ... you know, thing people want. **Swyx** [18:28] I don't think it's super answered here, but I have a theory also that reasoning models are better at context utilization because they can loop back. Normal autoregress models, they just kinda go left to right. But, uh, reasoning models, in theory, they can loop back and look for things that they needed connections for that they may not have paid attention to in, uh, in the initial pass. **Jeff Huber** [18:46] There's a paper today that showed, I think, maybe the opposite, but- **Swyx** [18:49] Really? **Jeff Huber** [18:50] I'll, I'll send it to you later. **Swyx** [18:51] Yeah. **Jeff Huber** [18:51] Yeah. **Swyx** [18:51] That'd be fascinating to figure out. **Alessio** [18:52] Yeah. There's papers every day. I thought the best thing was that you did not try to sell something. **Jeff Huber** [18:57] Yeah. **Alessio** [18:57] You were just like, "Hey, this thing is broken. Kinda sucks." How do you think about problems that you wanna solve versus research that you do to highlight some of the problems, and then hoping that other people will participate? Like- **Jeff Huber** [19:10] Mm **Alessio** [19:10] ... does everything that you talk about is on the Chroma roadmap, basically? Or are you just advising people, "Hey, this is bad, work around it, but don't ask us to fix it"? **Jeff Huber** [19:20] Kinda going back to what I said a moment ago, like, Chroma's broad mandate is to make the process of building AI applications more like engineering and less like alchemy. Um, and so, you know, that's a pretty broad tent, but we're a small team, and we can only focus on so many things, and we've chosen to focus very much on one thing for now. And so I don't think that... I don't have the hubris to think that we can ourselves solve this stuff conclusively for a very dynamic and large emerging industry. I think it does take a community. It does take, like, a rise in tide of people all working together. We intentionally wanted to, like, make very clear that, like, we do not have any, like, commercial motivations in this research. You know, we do not posit any solutions. We don't tell people to use Chroma. It's just, here's the, here's the problem. **Alessio** [20:03] It's implied. **Jeff Huber** [20:04] Um... **Swyx** [20:04] Yeah. **Jeff Huber** [20:06] Listen, we weren't sad that that was maybe an, maybe, maybe a positive implication- ... you know? But, like, still, there's still reasons around sp- you know, speed and cost regardless, I think, but there's just a lot of work to do. And I think that, like, it's interesting where, like, the labs don't really care, and they're not motivated to care. Increasing of the market to be, to be a good LLM provider, the main market seems to be consumer. You're just not that motivated to, like, help developers- **Alessio** [20:30] Like developers as a secondary concern. Yeah **Jeff Huber** [20:31] ... as a secondary concern. So you're just, like, not that motivated really to do the legwork to, like, help developers learn how to build stuff. **Alessio** [20:36] Yeah. **Jeff Huber** [20:36] And then, like, if you're a SaaS company or you're a consumer company, you're building with AI, you're, you know, an AI native company, like, this is your, like, this is your secret sauce. You're not gonna market how to do stuff. And so, like, I think there's just, like, a... There's a natural empty space, which is people that are actually have the motivations to, like, help show the way for how developers can build with AI. Like, there's just... There's not a lot of obvious people who are, like, obviously investing their time and energy in that, but I think that is obviously a good thing for us to do, and so that's kind of how I thought about it. **Swyx** [21:03] Just a bit of pushback on the consumer thing. Like, you say labs and... You know, don't, don't you think, like, OpenAI building memory into ChatGPT and making it available to literally everybody, and probably too much in your face, I would argue, but, like, they would really care to make the memory utilization good. I think context utilization and context engineering is important for them, too, even if they're only building for consumer and don't care about developers. **Jeff Huber** [21:26] Yeah, how good is it today is obviously one important question. **Swyx** [21:28] Yeah. **Jeff Huber** [21:29] Um, but we'll skip that one. Like, even if that's the case, are they actually gonna publish those findings? **Swyx** [21:33] No. Never. **Jeff Huber** [21:35] Exactly. It's alpha, right? Why would you give away your secrets? **Swyx** [21:38] Yeah. **Jeff Huber** [21:38] And so I think there's just, like, very few companies that actually are, like, in the position where, like, they have the incentive, and they really care about, like, trying to teach developers how to build useful stuff with AI. **Swyx** [21:48] Mm-hmm. **Jeff Huber** [21:48] And so I think that we have that incentive. ### Retrieval Strategies **Alessio** [21:50] But do you think you could get this to grow to the point of being the next needle in a haystack- **Jeff Huber** [21:54] Mm **Alessio** [21:54] ... and then forcing the models providers to actually be good at it? **Jeff Huber** [21:58] There's no path to forcing anybody to do anything, and so, uh, we, we thought about that when we were kind of putting this together. We were like, "Oh, maybe we should, like, sort of formulate this as a formal benchmark that you can make it very easy to, like..." We did open source all the code, so, like, you could... You know, if you're watching this and you're from a large model company, you can do this. You can take your new model that you haven't released yet, and you can run, you know, these numbers on it. And you know, I would rather have a model that has a 60,000 context, token context window that is able to perfectly pay attention to and perfectly reason over those 60,000 tokens than a model that's, like, 5 million tokens. Like, just as a developer, the former is, like, so much more valuable to me than the latter. I certainly hope that model providers do, like, pick this up as a thing that they care about and that they train around and that they, you know, evaluate their progress on and that they communicate to developers as well. Um, that would be great. **Alessio** [22:43] Do you think this will get bitter lesson as well? How do you decide which of the... Because you know, you're basically saying- **Jeff Huber** [22:48] Yeah **Alessio** [22:48] ... the models will not learn this. It's gonna be a, a trick on top of it that you won't get access to. **Jeff Huber** [22:54] I'm not saying that. **Alessio** [22:55] Well, but when you're saying that they will not publish how to do it well, it means that the model API will not be able to do it, but they will have something in ChatGPT- **Jeff Huber** [23:02] Mm **Alessio** [23:02] ... that will be able to do it. **Jeff Huber** [23:03] I see. **Alessio** [23:03] Yeah. **Jeff Huber** [23:04] It's very risky to bet what's gonna be a bitter lesson versus what is not. I don't think I'll hazard a guess. **Swyx** [23:10] Hopefully not AI engineers. **Jeff Huber** [23:12] Yeah. Hopefully not all of humanity. I don't know. You know. **Swyx** [23:14] Yeah. To me, also an interesting discipline developing, uh, just around context engineering, um, Lance Martin from LangChain did a really nice blog post with, like, all the different separations. **Jeff Huber** [23:25] Mm. **Swyx** [23:25] And then you in New York, you had... You hosted your, your first meetup. We're gonna do one here in San Francisco as well, but, uh, I'm just kinda curious, like, what are you seeing in the, in the fields? Like, who's doing interesting work? What are the top debates and that kind of stuff? **Jeff Huber** [23:38] I think this is still early. I mean, a lot of people are doing nothing. A lot of people are just still yeeting everything into the context window. That is very popular. **Swyx** [23:48] Yeah. **Jeff Huber** [23:48] And yeah, they're using context caching, and that certainly helps about, like, their cost and speed, but, like, isn't helping the context RAG problem at all. And so yeah, I don't, I don't know that there's lots of best practices in place yet. I mean, I'll, I'll highlight a few. So the, the problem fundamentally is quite simple. It's, you know, you have N number of sort of candidate chunks, and you have Y spots available, and you have to do the process to curate and cull down from 10,000 or 100,000 or a million candidate chunks, which 20? Matter right now. **Swyx** [24:22] Yeah. **Jeff Huber** [24:23] This exact step. **Swyx** [24:24] Mm-hmm. **Jeff Huber** [24:24] That optimization problem is not a new problem to many applications and industries, sort of a classic, um, a classic problem. And of course, like, what tools people use to solve that problem, again, I think it's still very early. Um, it's hard to say. But a few patterns that I've seen. So one pattern is to use what a lot of people call first stage retrieval to do a big call down. So that's to be using signals like vector search, like full text search, like metadata filtering, metadata search, and others to go from, let's say, ten thousand down to three hundred. Like we were saying a moment ago, like you don't have to give an LLM ten blue links. You can brute force a lot more. And so using an LLM as a re-ranker and brute forcing from three hundred down to thirty, I've seen now emerge a lot. Like, a lot of people are doing this. And it actually is, like, way more cost effective than I think a lot of people realize. I've heard of people that are running models themselves that are getting, like, a penny per million input tokens. **Swyx** [25:23] Wow. **Jeff Huber** [25:23] And, like, the output token cost is basically zero because it's like a, you know, the simplest- **Swyx** [25:28] These are dedicated re-ranker models, right? Not- **Jeff Huber** [25:30] No **Swyx** [25:30] ... full LLMs. **Jeff Huber** [25:31] No, these are LLMs. **Swyx** [25:32] Okay. **Jeff Huber** [25:32] They're just using LLMs as re-rankers. **Swyx** [25:34] Okay. **Jeff Huber** [25:34] And of course, there are also dedicated re-ranker models that by definition are gonna be still, like, cheaper because they're much smaller and faster 'cause they're much smaller. But, like, what I've seen emerge is, like, application developers who already know how to prompt are now applying that tool to re-ranking. And I think that, like, this is going to be the dominant paradigm. I actually think that, like, probably purpose-built re-rankers will go away in the same way that, like- **Swyx** [25:57] Wow **Jeff Huber** [25:57] ... purpose-built... I mean, they'll still exist, right? Like, if you, if you're at, if you're at extreme scale, extreme cost, yes. **Swyx** [26:02] Yeah. **Jeff Huber** [26:03] You'll care to optimize that. And the same with that if you're running with hardware, right? Like, you're just gonna use a CPU or a GPU unless you absolutely have to have an ASIC or an FPGA. And I think the same thing is true about, like, re-rankers, where, like, as LLMs become a hundred, a thousand times faster, a hundred, a thousand times cheaper, that, like, people are just gonna use LLMs for re-rankers. And that actually, like, brute forcing information curation is gonna become extremely, extremely popular. Now today, the prospect of running three hundred parallel LLM calls, even if it's not very expensive, you know, the tail latency on any one of those, like, three hundred LLM calls, API availability, like, it's all still really bad. And so, like, there are good reasons to not do that today in a production application, but those will also go away over time. So those patterns I think I've seen emerge that are... That, that's a, that is a new thing that I think I've only seen start to really become popular in the last few months. **Swyx** [26:57] Mm-hmm. **Jeff Huber** [26:57] And by popular, I mean, like, popular in, like, the leading tip of the spear, but I think will become a very, very dominant paradigm. **Swyx** [27:02] Yeah. We've, we've also covered a little bit on, especially on the code indexing side of this, the, the house. So everything we've t- been talking about applies to all kinds of contexts, but I think code is obviously a special kind of context- ### Code Indexing **Jeff Huber** [27:13] Mm-hmm **Swyx** [27:13] ... and corpus that you wanna index. We've had a couple of episodes, the cloud code guys and the, the client guys talk about they don't embed or they don't index your code base. They just give tools- **Jeff Huber** [27:23] Mm-hmm **Swyx** [27:23] ... and use the, use the tools of code search. **Jeff Huber** [27:25] Yeah. **Swyx** [27:25] And I've often thought about whether or not, like, this should be the primary context retrieval paradigm where, uh, when you build an agent, you, you effectively call out, uh, to another agent with all these sort of recursive re-rankers and summarizers or another agent with tools. **Jeff Huber** [27:40] Yep. **Swyx** [27:40] Um, or do you sort of glom them all into a single agent? I, I don't know if you have an opinion, obviously, 'cause agent is very ill-defined, but I'll just put it out there. **Jeff Huber** [27:48] Okay. Gotta pull that apart. So, you know- **Swyx** [27:49] Yeah **Jeff Huber** [27:50] ... indexing by definition is a trade-off. Like, when you index data, you're trading write time performance for query time performance. You're making it slower to ingest data, but much faster to query data, which obviously scales as data sets get larger. And so, like, if you're only grepping very small, you know, 15 file code bases, you probably don't have to index it, and that's okay. If you want to search all of the open source dependencies of that project, you all have done this before in VS Code or Cursor, right? You've, like, run a search over, like, the node modules folder. It takes a really long time to run that search. That's a lot of data. Like, so to make that indexed and sort of, again, make that trade-off of write time performance or query time performance, like that's what, that's what indexing is. Like, just to, like, demystify it, what is this, right? Like, that's what it is. You know, embeddings are known for semantic similarity today. Embeddings is just a generic concept of, like, information compression, and there's actually, like, many tools you can use embeddings for. I think embeddings for code are still extremely early and underrated, but Regex is obviously an incredibly valuable tool and, you know, we've actually worked on now inside of Chroma, both single load and distributed, we support Regex search natively. So you can do Regex search inside of Chroma because we've seen that as, like, a very powerful tool for code search. It's great. And we build indexes to make Regex search go fast at large data volumes. On the coding use case that you mentioned, another use case that... Another, another feature we added to Chroma is the ability to do forking. So you can take an existing index, and you can create a copy of that index in under 100 milliseconds for pennies. And in so doing, you then can just apply the diff for what file has changed to the new index. So any, like, corpus of data that's logically changing- **Swyx** [29:33] So very fast re-indexing is- **Jeff Huber** [29:35] Yeah, basically **Swyx** [29:36] ... the result. **Jeff Huber** [29:36] But now you can, like, have an index for, like, different... each commit. So if you wanna search different commits, search different branches or different release tags, like any corpus of data that's, like, logically versioned, you now can search all those versions very easily and very cheaply and cost effectively. And so, yeah, I think that, you know, that's kinda how I sort of think about, like, Regex and indexing and embeddings. I mean, yeah, the, the needle continues to move here. I think that anybody who claims to have the answer, you just, like, shouldn't listen to them. **Swyx** [30:03] When you say that code embeddings are underrated, what do you think that is? **Jeff Huber** [30:06] Most people just take generic embedding models that are trained on the internet- **Swyx** [30:10] Yeah **Jeff Huber** [30:10] ... and they try to use them for code. **Swyx** [30:12] Yeah. **Jeff Huber** [30:13] And, like, it works okay for some use cases, but does it work great for all use cases? I don't know. Another way to think about these different primitives and what they're useful for, fundamentally, we're trying to find signal. Text search works really well. Flexible search, text search works really well when the person who's writing the query knows the data. If I wanna search my Google Drive, I just, for the spreadsheet that has all my investors, I'm just gonna type in cap table, because I know there's a spreadsheet in my Google Drive called Cap Table. Full text search, great. It's perfect. I'm a subject matter expert in my data. Now, if you wanted to find that file and you didn't know that I had a f- spreadsheet called Cap Table, you're gonna type in the spreadsheet that has the list of all the investors. And of course, in embedding space, in semantic space, that's going to match. And so I think, again, these are just, like, different tools, and it depends on, like, who's writing the queries. It depends on what expertise they have in the data, like what blend of those tools is gonna be the right fit. My guess is that, like, for code today, it's something like 90% of queries or 85% of queries can be satisfactorily run with regex. Regex is obviously like the dominant pattern used by Google Code Search, GitHub Code Search. But you maybe can get, like, 15% or 10% or 5% improvement by also using embeddings. Very sophisticated teams also use embeddings for code as a part of their code retrieval code search stack. And, uh, you know, you shouldn't assume they just enjoy spending money on things unnecessarily. They're getting some... They're eking out some benefit there. And of course, like, for companies that want to be, like, top of their game and want to, like, you know, corner their market and want to serve their users the best, this is kind of what it means to build great software with AI. 80% is quite easy, but getting from 80% to 100% is where all the work is. And, like, you know, each point of improvement, like, is a point on the board and is a point that, like, I think users care about and is a point that you can use to, yeah, fundamentally just, like, serve your users better. **Swyx** [32:05] Do you have any thoughts on the developer experience versus agent experience? Like, this is another case where, well, we should maybe reformat and rewrite the code in a way that it's easier to embed and then train models there. Where are you on that spectrum? **Jeff Huber** [32:19] Yeah, I mean, one tool that I've seen work well for some use cases is instead of just embedding the code, you first have an LLM generate, like, a natural language description of, like, what this code is doing. And either you embed, like, just the natural language description, or you embed that and the code, or you embed them separately, and you put them in, like, separate, you know, vector search indexes. Chunk rewriting is kinda like the broad category for, like, what that is. Again, this is like, the idea here is, like, it's related to indexing, which is as much structured information as you can put into your write or your ingestion pipeline, you should. So all of the metadata you can extract, do it at ingestion. All of the chunk rewriting you can do, do it at ingestion. If you really invest in, like, trying to extract as much signal and kind of pre-bake a bunch of the signals at the ingestion side, I think it makes the downstream query task, like, much easier. But also, you know, just 'cause we're here, like, it's worth saying, like, people should be creating small golden datasets of what queries they want to work and what chunks should return, and then, like, they can quantitatively evaluate what matters. Maybe you don't need to do a lot of fancy stuff for your application. It's entirely possible that, again, just using regex or just using vector search, depending on the use case, that's maybe all you need. But I guess, again, anybody who's claiming to know the answer, you should... the first thing you should ask is, "Let me see your data." And then if they don't have any data, then you have your answer already. **Swyx** [33:47] I'll, uh, give a plug to a talk that you gave at the conference, uh, how to look at your data. Yes, looking at your data is, is important, having, having golden datasets. These are all, like, good practices that I feel like- **Jeff Huber** [33:56] Yeah **Swyx** [33:57] ... somebody should put into, like, a little pamphlet. Call it the Ten Commandments of AI Engineering or something. **Jeff Huber** [34:03] Okay. Might do that. Yeah. **Swyx** [34:04] Thou, thou shalt look at your data. **Jeff Huber** [34:06] Yeah. Yeah. ### Future Retrieval **Swyx** [34:07] We're about to move on to memory, but, like, I, I, you know, I wanna sort of leave space for, like, you know, any other threads that you feel like you always wanna get on a soapbox about that, yeah, that you, that we haven't covered. **Jeff Huber** [34:16] That's dangerous. That's a dangerous thing to ask. Um. **Swyx** [34:19] I have one to, to key off of. **Jeff Huber** [34:21] Okay. **Swyx** [34:21] Because I, I think, uh, I didn't, I didn't know, I, I didn't know where to insert this in the conversation, but we were kind of, uh, skirting near it, that I'm trying to explore. Which is, you know, uh, I think you, you had this rant about RAG, where the original transformer was sort of like an encoder-decoder architecture. **Jeff Huber** [34:37] Mm. **Swyx** [34:38] Then GBT turned, uh, most transformers into decoder-only. But then we're also encoding with all the, um, um, em- embedding models as encoder-only models. **Jeff Huber** [34:48] Mm. **Swyx** [34:48] So in, in some sense, we've sort of decoupled the transformer into first we encode everything with the encoder-only model, put it into a vector database like Chroma, and Chroma also does other stuff, but, you know. Um, then, then we decode with, uh, the LLMs. And I just think it's, like, a very interesting meta learning about- **Jeff Huber** [35:09] Yeah **Swyx** [35:10] ... the overall architecture. Like, it is stepping out of just the model to models and system. **Jeff Huber** [35:14] Mm. **Swyx** [35:15] And I'm curious if you have any reflections on that or if you have any modifications to what I just said. **Jeff Huber** [35:20] I think there's some intuition there, which is like the way we th- do things today is very crude and will feel very caveman in- **Swyx** [35:26] Yeah **Jeff Huber** [35:26] ... five or 10 years. You know, why aren't we just... why are we going back to natural language? Why aren't we just, like, passing the embeddings, like, directly to the models who are just going to functionally, like, re-put it into latent space, right? **Swyx** [35:37] Yeah, they have a very thin embedding, uh, layer. Yeah. **Jeff Huber** [35:39] Yeah. So I think, like, there's a few things that I think might be true about retrieval systems of the future. So, like, number one, they just stay in latent space. They don't go back to natural language. Number two, instead of doing, like... This is actually starting out to change, which is really exciting, but, like, for the longest time we've done one retrieval per generation. **Swyx** [35:55] Okay. **Jeff Huber** [35:55] So you retrieve, and then you stream out N number of tokens. Like, why are we not continually retrieving- **Swyx** [36:00] Yeah **Jeff Huber** [36:00] ... as we need to? Um- **Swyx** [36:02] Agentic RAG. **Jeff Huber** [36:03] Don't call it that. Stop. Stop. Um, but there was a paper or a paper in a, in a, you know, maybe like a GitHub that came out a few weeks ago, I think it was called, unfortunately, RAG R1, where they, like, teach, uh, DeepSeek-R1, you know, kinda give it the tool of how to retrieve. And so, like, kind of in its internal train of thought and its inference compute- **Swyx** [36:22] Yeah **Jeff Huber** [36:22] ... it's actually, like, searching as it's going. **Swyx** [36:23] There's also retrieval augmented language models. I think this is an older paper. **Jeff Huber** [36:26] Yeah, yeah. There's a bunch of, yeah, Realm and Retro, and there's kind of a long history here. **Swyx** [36:30] Yeah. **Jeff Huber** [36:30] Um, so I think that, you know, kind of- **Swyx** [36:32] Somehow not that popular. I don't know why. **Jeff Huber** [36:33] Somehow not that popular. Well- There— a lot of those have the problem where, like, either the retriever or the language model has to be frozen, and then, like, the corpus can't change, which most developers don't wanna, like, deal with the developer experience around. **Swyx** [36:46] I would say, like, we would do it if, if the gains were that high- **Jeff Huber** [36:50] Yeah **Swyx** [36:50] ... or- **Jeff Huber** [36:51] Yeah **Swyx** [36:52] ... the labs don't want you to do it. I don't know about... **Jeff Huber** [36:55] Yeah. **Swyx** [36:55] 'Cause the labs have a huge amount of influence. **Jeff Huber** [36:58] Labs have a huge amount of influence. I think it's also just, like, you don't get, you don't get points on the board by doing that well. You just, like, don't... No one cares. **Swyx** [37:04] Mm-hmm. **Jeff Huber** [37:05] The status games don't, don't reward you for solving that problem. So yeah, so broadly, continual retrieval I think will be interesting to see come onto the scene, number one. Number two, staying in embedding space will be very interesting. And then, yeah, there's some interesting stuff also about kind of like GPUs and how you're kinda like paging information into memory on GPUs that I think can be done, like, much more efficiently. Um- **Swyx** [37:26] Mm-hmm **Jeff Huber** [37:26] ... and this is more like 5 or 10 years in the future that we're kinda thinking about. But yeah, I think, I think when we look back and think this was, like, like, hilariously crude the way we do things today. **Swyx** [37:35] Maybe, maybe not, you know. We're solving IMO, uh, challenges with just language, you know. **Jeff Huber** [37:40] Yeah, it's great. **Swyx** [37:41] I'm still sort of working on the implications of that. Like, it's, it's still a, a huge achievement, but also very different than how I thought we would do things, like- **Alessio** [37:48] You said that memory is the benefit of context engineering. I think there's, uh, you, you had a rant on Twitter about stop making memory for AI so complicated. How do you think about memory? And is-- what, what are, like, maybe the other benefits of context engineering that maybe we were not connecting together? ### Memory **Jeff Huber** [38:07] I think memory is a good term. It is very legible to a wide population. Again, this is sort of just continuing the anthropomorphization of LLMs. You know, we ourselves understand how we are-- we as humans use memory. We're very good at... Well, some of us are very good at using memory to learn how to do tasks, and then those learnings being, like, flexible to new environments and, you know, the idea of being able to, like, take an AI, sit down next to an AI, and then instruct it for 10 minutes or a few hours, and kinda just, like, tell it what you want it to do, and it does something, and you say, "Hey, actually, do this next time," the same that you would with a human. At the end of that 10 minutes, at the end of those few hours, the AI be- is able to do it now, and the same level of reliability that a human could do it, like, is an incredibly attractive and exciting vision. I think that that will happen. And I think that memory, again, is like the... Memory is the term that, like, everybody can understand. Like, we all understand, our moms all understand. And, and, and the benefits of memory are also very appealing and very attractive. But what is memory under the hood? It's still just context engineering, I think, which is the domain of how do you put the right information into the context window. And so yeah, I think the memory is the benefit. Context engineering is the tool- **Alessio** [39:23] Yeah **Jeff Huber** [39:23] ... that gives you that benefit. And there may be stuff as well. I mean, may-maybe there's some version of memory where it's like, oh, you're actually, like, using RL to improve the model through data seen. And so I'm not su-suggesting that, like, only changing the context is the only tool which, you know, gives you great performance on tasks, but I think it's a very important part. **Alessio** [39:44] Do you see a big difference between synthesizing the memory, which is, like, based on this conversation, what is the- **Jeff Huber** [39:50] Hmm **Alessio** [39:50] ... implicit preference? **Jeff Huber** [39:52] Yeah. **Alessio** [39:52] That's one side, and then there's the other side, which is based on this prompt, what are the memories that I should put in? **Jeff Huber** [39:59] I think they will be all fed by the same data. So the same feedback signals that tell you how to retrieve better will also tell you what to remember better. So I don't think they're actually different problems. I think they're the same problem. **Swyx** [40:14] To me, the, the thing I'm wrestling with a little more is just, um, what are the structures of memory that make sense? So there's, like, obviously all these analogies with, like, long-term memory, short-term memory. Let us try to coin s-something around sleep. I do think that there, there definitely should be some sort of batch collection cycle, maybe sort of garbage collection cycle where, where it's like y- where the LLM is sleeping. But I don't know what makes sense. Like, we're making all these analogies based on- **Jeff Huber** [40:41] Hmm **Swyx** [40:41] ... what we think, how we think humans work. **Jeff Huber** [40:43] Yeah. **Swyx** [40:43] But maybe AI doesn't work the same way. **Jeff Huber** [40:45] Yeah. **Swyx** [40:46] I'm curious about, uh, anything that you've seen that, that's working. **Jeff Huber** [40:49] Yeah, I always, again, you know, as a through line of this conversation, I always get a little bit nervous when we start creating new concepts and new acronyms for things- **Swyx** [40:56] Yeah **Jeff Huber** [40:56] ... and then all of a sudden there's, you know, info charts that are like, "Here are the 10 types of memory," and you're like- **Swyx** [41:01] Sure **Jeff Huber** [41:01] ... why? These are actually... If you squint, they're all the same thing. Like, do they have to, do they have to be different, you know? Like- **Swyx** [41:06] You have to blow the people's minds. **Jeff Huber** [41:08] No, I, I don't think you do. I don't know. I think you gotta, you gotta resist the slot machine. The slot and the sloth machine. Um, compaction has always been a useful concept in- **Swyx** [41:19] Even in databases **Jeff Huber** [41:19] ... in databases on your computer. We all remember running defrag- **Swyx** [41:24] Mm-hmm **Jeff Huber** [41:24] ... on our Windows machines in a 1098. **Swyx** [41:26] Yeah. **Jeff Huber** [41:26] And, uh, you know, so yeah, again- **Swyx** [41:28] Some of us not old enough to do that. **Jeff Huber** [41:31] Mm-hmm. **Swyx** [41:31] I, I am though. **Jeff Huber** [41:33] Not at this table. Um, and, uh, yeah, so obvi-obviously offline processing is helpful, and I think that is also helpful in this case. And as we were talking about before, like, what is the goal of indexing? The goal of indexing is to, like, trade write time performance for query time performance. Compaction is another tool in the toolbox of, like, sort of write time performance. You're, you're reinvesting data. **Swyx** [41:53] Yeah. It's not indexing, but actually it is indexing in a way. **Jeff Huber** [41:55] It's sort of re-indexing, yeah. You're taking data, you're like, "Oh, maybe those two data points should be merged. Maybe they should be split. Maybe they should be, like, rewritten. Maybe there's new metadata we can extract from those." Like, let's look at the signal of how our application's performing. Let's try to figure out, like, are we remembering the right things or not? Like, the idea that there is going to be, like, a lot of offline compute and inference under the hood that helps make AI systems continuously self-improve is a sure bet. **Alessio** [42:19] What-- Part of the sleep time compute thing that we talked about was pre-computing answers. So based on the data that you have- **Jeff Huber** [42:26] Hmm **Alessio** [42:26] ... what are likely questions- **Jeff Huber** [42:28] Hmm **Alessio** [42:28] ... that the person is gonna ask- **Jeff Huber** [42:29] Yeah **Alessio** [42:29] ... and then can you pre-compute those things? **Jeff Huber** [42:31] Mm-hmm. **Alessio** [42:31] How do you think about that in terms of Chroma? **Jeff Huber** [42:34] We released a technical report maybe three months ago. The title is Generative Benchmarking, and the idea there is like, well, having a golden dataset is really powerful. Having a... What a golden dataset is, is you have a list of queries, and you have a list of chunks those queries should result in, and now you can say, "Okay, this retrieval strategy gives me, for these queries, gives me 80% of those chunks, whereas if I change the embedding model, now I get 90% of those chunks." That is better. And then you also need to consider cost and speed and API reliability and other factors, obviously, when making good engineering decisions. But, like, you can measure now- **Swyx** [43:10] Mm-hmm **Jeff Huber** [43:10] ... like, changes to your system. And so what we noticed was, like, developers had the data, they had the chunks, they had the answers, but they didn't have the queries. We did a whole technical report around how do you teach an LLM to write good queries from chunks? 'Cause again, you want, like, chunk query pairs, and so if you have the chunks, you need the queries. Okay, we can have a human do some manual annotation, obviously, but humans are inconsistent and lazy and, you know, QA is hard. And so can we teach an LLM how to do that? And, uh, so we sort of did a whole technical report and proved a strategy for doing that well. So I think generating QA pairs is really important for benchmarking a retrieval system, golden dataset. Frankly, it's also the same dataset that you would use to fine-tune in many cases. And so, yeah, there's definitely something, like, very underrated there. **Swyx** [43:58] Yeah. I'll throw a plus one on that. I think as much attention as the context rock paper is getting, I feel like generative benchmarking was a bigger aha moment for me, just 'cause I, I actually never came across the concept before. **Jeff Huber** [44:08] Mm. **Swyx** [44:08] And I think, like, actually more people will apply it to their own personal situations. Whereas context rot is just generally like, "Yeah, don't trust the models that much." **Jeff Huber** [44:16] Yeah. **Swyx** [44:16] But there's not much you can do about it, except do better context engineering. **Jeff Huber** [44:19] Yeah, yeah. Yes, yes. **Swyx** [44:20] Uh, whereas generative benchmarking, you're like, yeah, you generate your, your, your evals and, and, you know, uh, part of that is going... You- you're gonna need, uh, the datasets. And, uh, it'll, it'll sort of follow you into the place of all the better best practices that everyone advocates for. **Jeff Huber** [44:34] Mm-hmm. **Swyx** [44:34] Um, so. **Jeff Huber** [44:35] Yeah. **Swyx** [44:35] It's a very nice piece of work. **Jeff Huber** [44:37] I think having worked in applied machine learning developer tools now for 10 years, like, the returns to a very high quality, small label dataset are so high. **Swyx** [44:45] Mm-hmm. **Jeff Huber** [44:46] Everybody thinks you have to have, like, a million examples or whatever. No, actually, just, like, a couple hundred, even, like, high quality examples is extremely beneficial. **Swyx** [44:55] Yeah. **Jeff Huber** [44:55] And customers all the time, I say, "Hey, what you should do is say to your team, Thursday night we're all gonna be in the conference room, we're ordering pizza, and we're just gonna have a data labeling party for a few hours." And that, that's all it takes to bootstrap this. **Swyx** [45:08] Google does this, OpenAI does this, Anthropic does this. You're not above doing this. **Jeff Huber** [45:12] Great. **Swyx** [45:12] You know? Right. **Jeff Huber** [45:12] Yeah, exactly. **Swyx** [45:13] Yeah. **Jeff Huber** [45:14] Yeah. Look at your data. It's- Yeah, it's, again, it's what matters. **Swyx** [45:16] La- label... Maybe you should classify that as label your data, not look at. 'Cause look at seems a bit too- **Jeff Huber** [45:22] I agree with that, yeah. There's some more- **Swyx** [45:23] View only. **Jeff Huber** [45:24] Right. I agree with that. Yeah, yeah. Read and write. **Swyx** [45:27] Read and write. While you mentioned it, I should correct myself. It wasn't Standard Combination, it was Standard Cyborg. **Jeff Huber** [45:31] Mm. **Swyx** [45:32] My favorite fact about you is you're also a cyborg with your, with your leg. **Jeff Huber** [45:34] True. **Swyx** [45:35] Uh, there are people, if you see Jeff in person, you should ask him about it. Or maybe not. Maybe don't. I don't know. **Jeff Huber** [45:39] I don't care. Yeah. **Swyx** [45:39] Don't care. Uh, Standard Cyborg, Mighty Hive, and Know It. What were those lessons there that you're applying to Chroma? ### Lessons & Culture **Jeff Huber** [45:46] Yeah, more, more than I can count, um, I mean, it's a bit of a cliché, um, and it's very hard to be self-reflective and honest with yourself about a lot of this stuff. But I think viewing your life as being very short and kind of a, you know, a vapor in the wind, and therefore, like, only doing the work that you absolutely love doing, and only doing work that you... doing that work with people that you love spending time with and serving customers that you love serving is a very useful, like, North Star. Um, and you know, it may not be the North Star to, like, print a ton of money. **Swyx** [46:22] Yeah. **Jeff Huber** [46:22] In some sense. There may be faster ways to scam people into making $5 million or whatever. Um, so, but if I reflect on... And I'm happy to go into more detail obviously, but if I reflect on, like, my prior experiences, like, I was always making trade-offs. I was making trade-offs with, like, the people that I was working with, or I was making trade-offs with the customer that I was serving, or I was making trade-offs with, like, the technology and, like, how proud I was of it. And maybe it's sort of, like, an age thing, I don't know. But like, you know, the older that I get, I just more and more wanna do the best work that I can. And I want that work to not just be great work, but I also want it to be seen by the most number of people, because ultimately that is what impact looks like. You know, impact is not inventing something great and then nobody using it. Like, impact is inventing something great and as many people using it as possible. **Swyx** [47:01] Is any of that, uh, you know, and we can skip this question if it's sensitive, but, like, is any of that guided by religion, by Christianity? Uh, I... and I only ask this because I think you're one of a growing number of openly, outwardly- **Jeff Huber** [47:14] Yeah **Swyx** [47:15] ... positively religious people in the Valley. **Jeff Huber** [47:18] Yeah. **Swyx** [47:18] And I think that it's kind of what I wanna explore. You know, I'm not, I'm not, like, that religious myself, but I just kind of, like, how does that inform how you view your impact, your, you know, your choices that... There, there was a little bit of, of that in what you just said, but I wanted to sort of tease that out more. **Jeff Huber** [47:32] I think increasingly modern society is nihilist. Nothing matters. It's, uh, absurdist, right? Everything's a farce. Everything is power. **Swyx** [47:47] Everything's a comedy. **Jeff Huber** [47:48] Everything's a comedy. **Swyx** [47:49] A meme. Yeah. **Jeff Huber** [47:50] Yeah, exactly. And so, like, it's very rare, and I'm not saying that I always am the living exemplar of this, but, like, it's very rare to meet people that have genuine conviction about what flourishing for humanity looks like, and that it's very rare to meet people that are, like, actually willing to sacrifice a lot to, like, make that happen, and to start things that, like, they may not actually see complete in their lifetimes. Like, it used to be commonplace that people would start projects that would take centuries to complete. **Swyx** [48:20] Yeah. **Jeff Huber** [48:20] And, you know, now that's, like, less and less the case. **Swyx** [48:24] The image that comes to mind is the Sagrada Família in Barcelona. **Jeff Huber** [48:28] Right. **Swyx** [48:28] Which, uh, I think was started like 300 years ago, and it's completing next year. **Jeff Huber** [48:32] Yeah. I've seen it in construction, but I can't wait to see it completed- **Swyx** [48:36] Next year **Jeff Huber** [48:36] ... as well. **Swyx** [48:37] Yeah, I'm sure the places are booked out already. **Jeff Huber** [48:39] Yeah. **Swyx** [48:39] Yeah. **Jeff Huber** [48:40] And so, you know, it's, it's common... There are actually, you know, a lot of, like, religions in Silicon Valley. I think AGI is also a religion. **Swyx** [48:46] Yeah. **Jeff Huber** [48:47] It has a problem of evil. We don't have enough intelligence. It has a solution, a deus ex machina. It has the second coming of Christ, that AGI, the singularity is going to come. It's going to save humanity, 'cause we will now have infinite and free intelligence. Therefore, all of our problems will be solved and, uh, you know, we will live in sort of, like, the palm of grace for all eternity. It's gonna solve death, right? And so, like, I think that, like, religions still exist in Silicon Valley. I think that it's like, you know, there's a conservation of religion. You kind of can't get rid of it. **Swyx** [49:15] Yeah. **Jeff Huber** [49:15] Um, but- **Swyx** [49:16] The God gene **Jeff Huber** [49:17] Yeah, I mean, you know, people have different terms for this, but, uh, like I think that I'm always skeptical of religions that haven't been around for more than five years. Put it that way. **Swyx** [49:27] Yeah, this is survivorship bias. Anyway, I, I, I do think like you're one of the more prominent ones that I, that I know of, and, uh, I think you, you guys are a force for good, and, uh, I, I like to encourage more of that. I don't know. Uh, you know, s- people should believe in something bigger than themselves and- **Jeff Huber** [49:42] Yeah **Swyx** [49:42] ... build for, uh, s- plant trees under which they will not sit. Um, it's, it's, I, I... Is, am I mangling the quote? Is that a, is that actually a biblical quote? **Jeff Huber** [49:50] I don't think that's a biblical quote, but I like that quote. **Swyx** [49:52] Yeah, yeah. **Jeff Huber** [49:52] That's a good one. **Swyx** [49:52] Yeah, yeah. **Jeff Huber** [49:53] So yeah. Plus one. **Swyx** [49:54] Like, uh, I think society is really collapsed when, like, you just live for yourself. That, that, that really is true. **Jeff Huber** [49:59] Agreed. **Alessio** [50:00] Who does your design? Because, uh, all of your swag is great. Your office looks great. The website looks great. The docs look great. How much of that is your input? How much of that do you have somebody who just gets it, and how important is that to, like, making the brand- **Jeff Huber** [50:16] Mm **Alessio** [50:17] ... part of the culture? **Jeff Huber** [50:18] I think all value... You know, again, going back to C- the Conway's Law thing, like you ship your org chart, you ship what you care about as a founder in some sense, and like I do care deeply about this aspect of what we do. And so I think it does, it does, it does come from me in some sense. Um, I can't take at all credit for everything we've done. We've had the opportunity to work with some, like, really talented designers, and we're hiring as well for that. So if people, you know, are listening to this and want to apply, please do. I think, I mean, it's cliche to, uh, to crib, um, Patrick Collison quotes, but he does seem to be one of the, like, most sort of public embodies of this idea that, like, how you do... And I'm not sure this is a direct quote from him. To be clear, this is more of just a broad aphorism, but, like, how you do one thing is how you do everything. And just ensuring that there's a consistent experience of what we're doing, where, like you said, if you come to our office, like it feels intentional and thoughtful. If you go to our website, it feels intentional and thoughtful. If you use our API, it feels intentional and thoughtful. If you go through our interview process, it feels intentional and, and purposeful. I think that's so easy to lose. It's just so easy to, like, lose that. And in some ways, the only way that you keep that is by insisting on that that standard remain. And I think that that is, like, one of the main things that, like, I can do really for the company, like, as a leader. It's sort of cringe to say, but, like, you do kind of have to be like the curator of taste. **Swyx** [51:47] Yeah. **Jeff Huber** [51:47] It's not that I have to stamp everything that goes out the door before it does, but at a minimum, companies... You know, maybe it's not even, like, downhill in quality. It's not sort of legible that any one thing is bad or worse, but it's li- more like people just have their own expressions of what good looks like and, like, you know, they turn that up to 11, and then, like, the brand becomes incoherent. Like, what does this thing mean, and like what do they stand for? Again, there's not... There's, there's no longer a single voice. Yeah, I don't... Again, I'm not claiming that I'm good at, perfect at this or good at this, but we certainly... We, we, we, we wake up every, we wake up every day and we try. **Swyx** [52:20] You have a lot of, uh... It's very powerful the, the, the skill you have to convey like straightforward principles and values and thoughtfulness, I think, in, in everything that you do. Like, yeah, I, you know, I... You know, you know I've, I've been impressed with your work for a while. **Jeff Huber** [52:35] Thank you. **Alessio** [52:36] Anything we're missing? You're hiring designers. Any other roles that you have open that you want people to, to apply for? **Jeff Huber** [52:43] If you're a great product designer, um, that wants to work on developer tools, I think we have one of the most kind of unique opportunities at Chroma. If you are interested in extending the kind of research that we do, that's also an interesting opportunity. We're always also hiring, like, very talented engineers that wanna work with other people that are very passionate about kind of low-level distributed systems and in some ways solving all the hard problems so that application developers don't have to. **Swyx** [53:08] When you say that, can you double-click on low-level distributed systems? People always say this, and then like, okay, Rust, like- ... like, you know- **Jeff Huber** [53:16] Yeah **Swyx** [53:16] ... like Linux kernel. What are we talking here? **Jeff Huber** [53:18] Yeah. I mean, like- ... the, maybe, like, you know, a useful encapsulation of this is, like, if you care deeply about things like Rust or deterministic simul-simulation testing or- **Swyx** [53:32] Raft, Paxos **Jeff Huber** [53:33] ... TLA+ consensus. **Swyx** [53:34] TLA+ really? **Jeff Huber** [53:36] Um- **Swyx** [53:36] Wow. **Jeff Huber** [53:37] You know, if you just keep... I'm saying these are, these are like proxies for- **Swyx** [53:40] Okay **Jeff Huber** [53:40] ... you would like, you would like the work that we do here. **Swyx** [53:42] I just really wanna tease out the hiring message, but also I, um, part of my goal is also to try to identify who are, what, what is the type of engineer that people, that startups are really trying to hire and they cannot get. **Jeff Huber** [53:52] Hm. **Swyx** [53:52] Because the, the better we can identify this thing, I can, you know, maybe create like some kind of branding around it, create an event, and, like- **Jeff Huber** [53:59] Yeah **Swyx** [53:59] ... get these... Like, there's a, there's a supply side and a demand side, and they can't find each other. **Jeff Huber** [54:04] Yeah. **Swyx** [54:04] And that's why I put AI engineer together was that that was- **Jeff Huber** [54:07] Yeah **Swyx** [54:07] ... that was part of it. **Jeff Huber** [54:07] Yeah, yeah. **Swyx** [54:08] But then this distributed systems person, which, like, I have heard from you and, like, 100 other startups. **Jeff Huber** [54:14] Mm. **Swyx** [54:14] What is the skill set? What are they called? **Jeff Huber** [54:16] Yeah. **Swyx** [54:16] What do they do? And part of that is, like, it's... Part of that is cloud engineering because a lot of times you're just dealing with AWS. **Jeff Huber** [54:22] Sure. **Swyx** [54:22] A lot of that, a lot of times you're just dealing with, I don't know, debugging network calls and, and, uh- **Jeff Huber** [54:28] Mm-hmm **Swyx** [54:28] ... consistency things if you're doing replication or whatever. **Jeff Huber** [54:30] Mm-hmm. **Swyx** [54:31] Um, where do they go? What do they do? Yeah, yeah, like, but- **Jeff Huber** [54:34] Yeah **Swyx** [54:34] ... they don't use TLA+ at work, you know? **Jeff Huber** [54:36] Probably not, yeah. I mean, last year I started, like, the SF Systems Group. **Swyx** [54:41] Yes. **Jeff Huber** [54:41] Sort of- **Swyx** [54:41] The reading group. **Jeff Huber** [54:42] Um, yeah, there was like presentations and- **Swyx** [54:44] Yeah **Jeff Huber** [54:44] ... like the point of that was like, let's bring, let's create a meeting place for the, like, care about this topic. **Swyx** [54:50] Yeah, yeah. **Jeff Huber** [54:50] Because, like, there wasn't really a place in the Bay Area for people to do that. **Swyx** [54:54] Yeah. **Jeff Huber** [54:55] Um, so that continues to go now and continues to run, which is great. I mean, to be clear, we have a lot of people on the team who are extremely good at this, and so, like, it's not that we have zero, it's that we have six or seven. Um- **Swyx** [55:05] And you want 20. **Jeff Huber** [55:06] But well we... Yeah, we don't... Yeah. It's not that we want more, but we are in some ways... Like, I feel like our product roadmap is very obvious, and we know exactly what we need to build for the next even, like, 18 months. But quality is always a limiting function. Quality and focus are always limiting functions, and like, while, yes, I will always make my land acknowledgement to Mythical Man-Month, eventually, like- **Swyx** [55:30] Let's get more people. **Jeff Huber** [55:31] You do, you kind of do need more people because you need more focus. Like, you need more people to care deeply about the work that they do. And the AI is certainly an accelerant. It's helpful, and it's the reason that our team is still very small today relative to many of our competitors is because, like, I think we've really embraced, like, those tools. **Swyx** [55:47] Your cursor shop, Cloud Code, Windsurf. **Jeff Huber** [55:49] People use whatever they want. **Swyx** [55:50] Okay. **Jeff Huber** [55:51] Yeah. So I think all of those tools get some usage internally. So far, uh, we've still not found that really any AI coding tools are particularly good at Rust though. Um- **Swyx** [56:00] Hmm **Jeff Huber** [56:00] ... I think I'm not sure why that is other than the obvious. There's just like not that many examples of great Rust on the internet, and so, um, you know. Yeah. **Swyx** [56:09] Yeah. **Jeff Huber** [56:09] But- **Swyx** [56:09] You, you would think that, you know, Rust, uh, errors would be help you debug it- itself, right? **Jeff Huber** [56:17] You would think that. **Swyx** [56:17] Apparently not. Okay. All right. I have zero experience in that, in that front. **Jeff Huber** [56:21] Yeah. **Swyx** [56:22] Uh, I have, I've contributed three things to the Rust SDK of Temporal, and that was my total experience with Rust. But, uh, I, I think it's definitely on the rise. It's, it's Zig, it's Rust. **Jeff Huber** [56:32] Mm. **Swyx** [56:32] And I, I don't know if there's a third cool languages. Like- **Jeff Huber** [56:36] I think Ghost accounts **Swyx** [56:36] ... Go- Golang, yeah. **Jeff Huber** [56:37] Yeah. Ghost accounts. **Swyx** [56:38] If you're in, in those, in that, in that bucket, uh, reach out to Jeff. But, uh, otherwise, I think we're good. **Alessio** [56:44] Thanks for coming on. **Jeff Huber** [56:45] Thanks for having me, guys. Good to see you. **Swyx** [56:47] Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.