# RAG is a hack - with Jerry Liu of LlamaIndex

Latent Space · 2023-10-12

<https://addtry.com/eb5c8c2b-f9e8-4c83-9912-ba2353fb27d6>

In this episode of Latent Space, Jerry Liu of LlamaIndex argues that while RAG is fundamentally a hack, it remains the most practical approach for grounding LLMs with external data, and his open-source framework now serves 600,000 monthly downloads. Liu recounts how LlamaIndex originated from a hackathon at Robust Intelligence, evolving from an experimental tree-index to a modular toolkit spanning data loading, retrieval, synthesis, and agent loops. He explains the trade-offs between RAG and fine-tuning, noting that context window limits force algorithmic rather than learned optimizations, but RAG offers transparency and access control that fine-tuning cannot. Liu details the company’s $8.5M Greylock raise, the launch of LlamaHub for community-contributed data loaders, and the open-source SEC Insights app as a production-grade template. Looking ahead, he emphasizes the need for better retrieval benchmarks and predicts that future personalization will move beyond vector stores into model-internal memory architectures.

## Questions this episode answers

### Why does Jerry Liu consider RAG a hack?

Jerry Liu says RAG is a hack because it keeps the model fixed and uses algorithmic methods to stuff retrieved text into the prompt, rather than optimizing the full system end-to-end through machine learning. He calls it a 'very good hack' since it is easy to use and works well enough for most applications, but from a pure optimization standpoint, it leaves out learnable components that could improve performance.

[24:49](https://addtry.com/eb5c8c2b-f9e8-4c83-9912-ba2353fb27d6?t=1489000)

### How did the LlamaIndex project originate?

LlamaIndex began as a side project after an internal hackathon at Robust Intelligence in October 2022. Jerry Liu was testing GPT-3 but faced context window limits, leading him to build a tree-index to organize and traverse information. Initially a thought exercise, it gained traction as developers sought ways to apply LLMs to personal data, eventually evolving into the modular tool it is today.

[6:15](https://addtry.com/eb5c8c2b-f9e8-4c83-9912-ba2353fb27d6?t=375000)

### How does Jerry Liu see the future of fine-tuning versus RAG?

Liu believes RAG is currently the default because it is easier to adopt and meets most needs, but he anticipates fine-tuning will eventually take over some RAG capabilities. He argues that a machine learning–based approach could optimize the entire system end-to-end, while current API limitations and costs make it impractical. He foresees a future where models may bake in retrieval-like behavior, but RAG remains advantageous for transparency and access control.

[27:26](https://addtry.com/eb5c8c2b-f9e8-4c83-9912-ba2353fb27d6?t=1646000)

## Key moments

- **[0:00] Intro**
- **[5:29] Origins**
  - [5:29] Robust Intelligence's October 2022 hackathon gave birth to both LangChain and LlamaIndex.
- **[11:36] Founding**
  - [14:01] RAG remains necessary despite 100K context windows due to data scale and transfer costs.
  - [15:46] Jerry Liu raised $8.5M from Greylock in a rapid 3-day fundraising process.
  - [17:52] Jerry Liu's co-founder Simon, from Uber ATG, was chosen for superior skills and shared grit.
  - [19:58] LlamaIndex reached 600K monthly downloads, 10K Discord members, and 50% GitHub star growth in 3 months.
- **[20:10] Context & RAG**
- **[24:27] RAG vs Fine-tuning**
  - [24:29] Jerry Liu: 'RAG is basically just a hack that turns out it's a very good hack.'
  - [25:01] Jerry Liu: RAG is purely algorithmic; the model is fixed, retrieval is hand-tuned heuristics.
  - [28:04] RAG provides transparency with source citations and access control for different data permissions.
  - [29:40] Jerry Liu predicts fine-tuning will gradually replace some RAG tasks as architectures evolve.
  - [30:25] Q: Can fine-tuning inject knowledge into LLMs? Jerry's OpenAI fine-tuning attempt didn't work.
- **[34:44] Inside LlamaIndex**
  - [34:44] Q: Why build RAG from scratch? Jerry Liu: it builds intuition for debugging and optimization.
  - [37:08] LlamaIndex's upcoming tutorials will expose RAG internals beyond the 3-line quick start.
  - [39:29] LlamaIndex's modular components: data loaders, retrievers, synthesizers, and agent reasoning.
  - [41:47] LlamaIndex coined 'data agents' for LLM-powered reasoning chains over structured data.
- **[43:10] Optimizing RAG**
  - [44:45] Query-side embedding transforms avoid full re-indexing when fine-tuning, suggests Joe from Vespa.
  - [46:01] Q: How to handle stale or duplicate documents in RAG? LlamaIndex explores time-weighted retrieval.
  - [48:08] LLM query decomposition improves retrieval accuracy by breaking questions into subqueries.
  - [48:56] RAG evaluation needs benchmarks to compare retrieval techniques, Jerry Liu argues.
- **[49:11] Production**
  - [49:23] SEC Insights: LlamaIndex's open-source full-stack chatbot for SEC filings demonstrates production RAG.
  - [52:35] LlamaIndex's planned managed service offers one-line toggle for monitoring, evals, and enterprise features.
- **[54:33] Ecosystem**
  - [54:34] LlamaHub community contributions include PDF, Gmail, Notion data loaders; building quality connectors is hard.
  - [57:25] OpenAI models lead in LlamaIndex usage, but Llama 2 open-source adoption is growing fast.
  - [59:27] Q: Are RAG-specific models viable? Jerry Liu doubts they can outpace rapid LLM advancements.
  - [1:02:21] The next vector DB frontier combines structured SQL and semantic search in one system.
  - [1:04:15] LlamaIndex explores graph-based data relationships to power richer LLM reasoning.
- **[1:04:36] Evals & Agents**
  - [1:04:54] Q: How to evaluate RAG? Start with end-to-end checks, then use retrieval benchmarks like ranking metrics.
  - [1:05:37] Jerry Liu: GPT-4-based synthetic evals will become the scalable standard for RAG benchmarks.
  - [1:07:46] LlamaIndex generates synthetic question-context pairs with GPT-4 for retrieval evaluation.
  - [1:08:02] Evaluating data agents is more complex due to multi-step loops and inter-agent communication.
- **[1:09:20] Lightning Round**
  - [1:09:23] Jerry Liu: 'just the ability of LLMs to generate believable outputs' happened faster than expected.
  - [1:10:22] Personalized memory will eventually be baked into model architecture, not just external RAG, predicts Jerry Liu.
  - [1:11:04] Swyx vs Jerry Liu debate: Can neural net weights reliably store personalized memory versus vector databases?
  - [1:11:42] Jerry Liu's takeaway: explore SEC Insights and build RAG from scratch for deeper engineering intuition.

## Speakers

- **Alessio** (host)
- **Swyx** (host)
- **Jerry Liu** (guest)

## Topics

Agent Platforms

## Mentioned

Anthropic (company), Cohere (company), Greylock (company), Meta (company), OpenAI (company), Quora (company), Robust Intelligence (company), Chroma (product), GPT-3 (product), GPT-4 (product), LanceDB (product), LangChain (product), Llama (product), LlamaHub (product), LlamaIndex (product), Ollama (product), SEC Insights (product), Streamlit (product), Weaviate (product), pgvector (product)

## Transcript

### Intro

**Alessio** [0:07]
Hey, everyone. Welcome to the Latent Space Podcast. This is Alessio, partner and CTO in residence at Decibel Partners, and I'm joined by my co-host, Swyx, founder of Small AI.

**Swyx** [0:17]
And today, we finally have Jerry Liu on the podcast. Hey, Jerry.

**Jerry Liu** [0:21]
Hey, you guys. Hey, Swyx and Alessio. Thanks for having me.

**Swyx** [0:24]
It's so weird because we keep running into each other in San Francisco AI events, so it's kinda weird to finally just have a conversation recorded for everybody else.

**Jerry Liu** [0:32]
Yeah, no. I'm really looking forward to this. I'm excited for the questions.

**Swyx** [0:36]
Um, so I tend to introduce people on their formal background, and then, um, ask something on the more personal side. So you are part of the Princeton gang, um .

**Jerry Liu** [0:46]
Yeah. I, I don't know if there is, like, an official, official Princeton gang, but-

**Swyx** [0:50]
There is an official Princeton gang.

**Jerry Liu** [0:50]
Okay.

**Swyx** [0:50]
I, I, I attended your meeting. There was, like-

**Jerry Liu** [0:52]
Oh

**Swyx** [0:52]
... four of you.

**Jerry Liu** [0:53]
Oh, cool. Okay. Nice.

**Swyx** [0:54]
With Prem and, and the others.

**Jerry Liu** [0:55]
Oh, yeah, yeah, yeah.

**Swyx** [0:56]
Yeah.

**Jerry Liu** [0:56]
Yeah.

**Swyx** [0:57]
Um, well, you did bachelor's in CS and a certificate in finance. That's also fun. Um, I also, uh, did finance, and I think I saw that you also interned at Two Sigma, where I worked in New York.

**Jerry Liu** [1:06]
Mm-hmm.

**Swyx** [1:07]
You were a machine learning engineer-

**Jerry Liu** [1:09]
You were at Two Sigma?

**Swyx** [1:09]
Yeah.

**Jerry Liu** [1:09]
Oh.

**Swyx** [1:09]
Very briefly.

**Jerry Liu** [1:10]
Oh, cool. All right.

**Swyx** [1:11]
Yeah.

**Jerry Liu** [1:11]
I didn't know that. Okay.

**Swyx** [1:11]
Uh, that was my first, like, proper engineering job before I went into dev rel.

**Jerry Liu** [1:15]
Oh, okay.

**Swyx** [1:15]
Yeah.

**Jerry Liu** [1:15]
Oh, wow. Nice.

**Swyx** [1:16]
Um, and then you were a machine learning engineer at Quora, um, um, AI research scientist at Uber for three years, and then two years machine learning engineer at Robust Intelligence before starting LlamaIndex.

**Jerry Liu** [1:27]
Mm-hmm.

**Swyx** [1:28]
So that's your LinkedIn. What's not on your LinkedIn that people should know about you?

**Jerry Liu** [1:31]
I think back during my Quora days-

**Swyx** [1:33]
Yeah

**Jerry Liu** [1:33]
... um, I had this, like, thor- three-month phase where I just wrote, like, a ton of Quora answers. And so I think if you look at my, my tweets nowadays, you could basically see that as, like-

**Swyx** [1:42]
Oh

**Jerry Liu** [1:42]
... the V2 of my, my three-month, like, Quora stint where I just, like, went ham on Quora for a bit. Um, I actually... I think I was... Back then, actually, uh, when I was working at Quora, I think the thing that everybody was fascinated in was, um, just, like, general, like, deep learning advancements and stuff like GANs and, and generative, like, images and, and just, like, new architectures that were evolving, and it was a pretty exciting time to be a researcher, actually, 'cause you were going in, like, really understanding some of the new techniques.

So I kinda used that as, like, a learning opportunity, basically, just, like, read a bunch of papers and then answer questions on Quora.

**Swyx** [2:12]
Cool.

**Jerry Liu** [2:12]
Uh, and so you can kinda see traces of that basically in my current Twitter, where it's just, like, really about kind of, like, framing concepts and trying to make it understandable and educate other users on it.

**Swyx** [2:21]
Yeah. I've said... So a lot of people come to me for my Twitter advice, but, like, I think you are doing one of the best jobs in AI Twitter, uh, just explaining concepts and just consistently getting hits out.

**Jerry Liu** [2:31]
Thank you.

**Swyx** [2:32]
Um, and I, I did know it was due to the Quora training. Um, just a side note on Quora.

**Jerry Liu** [2:36]
Mm-hmm.

**Swyx** [2:37]
A lot of people, including myself, like, kind of wrote off Quora as, like, one of the Web 1.0, like, sort of question answer forums, but, uh, now I think it's becoming... seeing a resurgence, obviously-

**Jerry Liu** [2:46]
Mm-hmm

**Swyx** [2:46]
... due to Poe.

**Jerry Liu** [2:46]
Mm-hmm.

**Swyx** [2:47]
Um, and obviously, Adam D'Angelo has always been a leading tech figure, but-

**Jerry Liu** [2:50]
Mm-hmm

**Swyx** [2:50]
... what do you think is, like, kind of underrated about Quora?

**Jerry Liu** [2:52]
I really liked the mission of Quora when I, when I joined. Um, in fact, um, I think when, um, I interned there, like, in 2015, and then I joined full-time in 2017, one is, like, they had and, and, and they have, like, a very talented engineering team, um, and, and just, like, really, really smart people.

And the other part is the whole mission of the company is to just, like, spread knowledge and to educate people.

**Swyx** [3:13]
Mm-hmm.

**Jerry Liu** [3:13]
Um, right? And, and to me, that really resonated. I really liked the idea of just, like, education and democratizing the flow of information. And if you imagine, like, um, kinda back then, it was like, okay, you have Google, which is, like, for search, but then you have Quora, which is just, like, user-generated, like, grassroots type content.

And I really liked that concept because it's just like, okay, there's certain types of information that aren't accessible to people, but you can make accessible by just, like, surfacing it. And, and so actually, I don't know if, like, most people know that about, like, Quora, like, and, and w- if they've used the product, whether through, like, SEO, right, or, or kind of, like, actively, but that really was what drew me to it.

**Swyx** [3:46]
Yeah. I, I think most people challenges with it is that sometimes you don't know if it's, like, a veiled product pitch, right?

**Jerry Liu** [3:52]
Yeah.

**Swyx** [3:52]
It's like, you know.

**Jerry Liu** [3:53]
Of course, like, quality of the answer matters quite a bit, and then-

**Swyx** [3:55]
It's like five alternatives, and then here's the one I work on.

**Jerry Liu** [3:57]
Yeah, like- ... recommendation issues and, and all that stuff. I just... I worked on Rexus at Quora-

**Swyx** [4:01]
Okay

**Jerry Liu** [4:01]
... actually, so, so I got-

**Swyx** [4:02]
So how do you solve stuff like that?

**Jerry Liu** [4:03]
... a taste of some of that stuff.

**Swyx** [4:04]
Yeah.

**Jerry Liu** [4:04]
Well, I mean, I kind of mo- more approached it from, um, machine learning techniques, which might be a ni- nice segue into RAG, actually. A lot of it was just information retrieval. We weren't, like, solving anything that was, like, super different than what was standard in the industry at the time, but just, like, ranking based on user preferences.

Um, I think a lot of Quora was very metrics driven, so just, like, trying to maximize, like, you know, daily active hours, like, you know, um, try- time spent on site, those types of things. Um, and all the machine learning algorithms were really just based on embeddings.

Um, you know, you have a user embedding, and you have, like, item embeddings, and you try to train the models to try to maximize the similarity of these. Uh, and it's basically a retrieval problem.

**Swyx** [4:40]
Okay.

**Jerry Liu** [4:40]
Yeah.

**Swyx** [4:40]
So you've been working on RAG for longer than most people think?

**Jerry Liu** [4:43]
Uh, well, well, kind of. So it's- ... I worked there for, like, a year, right?

**Swyx** [4:46]
Yeah.

**Jerry Liu** [4:46]
Just, uh, transparently. And then I worked, um, at Uber, where I was not working on ranking. It was more, um, like, kind of deep learning training for, for self-driving and computer vision and that type of stuff. Um, but I think, yeah, I mean, I think in the LLM world, it's kind of just like a combination of, like, everything these days.

I mean, retrieval's not really LLMs, but, like, it's, it's, it fits within the space of, like, LLM apps. Um, and then obviously, like, having knowledge of the underlying deep learning architectures helps. Having knowledge of basic software engineering principles helps, too.

Um, and so I think ni- uh, it's kinda nice that, like, this whole LLM space is basically just, like, combination of just, like, a bunch of stuff that, uh, you probably ha- like people have done in the past.

**Swyx** [5:24]
It's good. It's like a summary capstone project.

**Jerry Liu** [5:27]
Yeah, exactly. Yeah.

**Swyx** [5:29]
Um, yeah.

### Origins

**Alessio** [5:29]
And, and before we dive into, um, LlamaIndex, what did they feed you at Robust Intelligence that both you and Harrison- ... from LangChain came out of it at the same time? Was there, like- Yeah. Is there any fun story of, like, how both of you kinda came up with kinda like core infrastructure to LLM workflows today?

Or how close were you at Robust? Like any, any fun behind the scenes?

**Jerry Liu** [5:53]
Yeah, yeah. We, um, we, we worked pretty closely. I mean, we were on the same team for like two years. I got to know Harrison and the rest of the team pretty well. I mean, I, I highly respect the people there.

The people there were very driven, very passionate, and it definitely pushed me to be, you know, a better engineer and leader and those types of things. Um, yeah, I don't really have a concrete explanation for this. Um, I think it's more just we have, like, an LLM hackathon, uh, around, like, September.

Uh, this was just like exploring GPT-3, or it was October actually. And then the day after, I went on vacation for a week and a half, um, and so I just didn't track Slack or anything. Um, came back, saw that Harrison started LangChain.

I was like- ... "Oh, that's cool." I was like, "Oh, I'll play around with, uh, LLMs a bit," and then hacked around on stuff. And I think I've told this story a few times, but, you know, I was, like, trying to feed in information into, uh, the GPT-3 and then, then you deal with, like, context window limitations, and, uh, there was no tooling or really practices to try to understand how do you, you know, get GPT-3 to navigate li- large amounts of data, and that's kinda how the project started.

Really was just one of those things where, uh, early days, like, we were just trying to build something that was interesting, um, and not really... I, like, I wanted to start a company. Um, I had other ideas actually of what I wanted to start.

Um, and, and I was very interested in, for instance, like multi-modal data, like video data and that type of stuff. Uh, and then this just kinda grew and eventually took over the other idea.

**Swyx** [7:10]
Text is the universal interface.

**Jerry Liu** [7:13]
I think so. I think so. I actually think once the multi-modal models come out, I think there's just, like, mathematically nicer properties if you can just get, like, joint multi-modal embeddings, like clip, clip style. Um, but how, like, text is really nice because from a software engineering principle, it just makes things way more modular.

You just convert everything into text, and then you just represent everything as text.

**Swyx** [7:31]
Yeah.

**Jerry Liu** [7:32]
Yeah.

**Swyx** [7:32]
I'm just explaining retroactively why working on LlamaIndex took off versus-

**Jerry Liu** [7:36]
Yeah. Yeah, yeah

**Swyx** [7:36]
... if you had chose to spend your time on multi-modal, we would, probably wouldn't be talking about whatever you, you ended up working on.

**Jerry Liu** [7:42]
Yeah. No, that's true.

**Swyx** [7:43]
It, it, it struggled. Um, yeah, I, I think so. Um, so i- interesting. So, um, on November, so on November 9th, so that was a very productive month I g- I guess. So October, November. November 9th, you announced GPT-3 Index, and you picked a tree logo.

Very, very, very cool. Everyone, every project must have an emoji.

**Jerry Liu** [8:01]
Yeah, yeah. That, that probably was somewhat inspired by LangChain, but I, I will admit, yeah.

**Swyx** [8:07]
It uses GPT to build a knowledge tree in a bottom-up fashion by applying, applying a summarization prompts for each node.

**Jerry Liu** [8:12]
Yep.

**Swyx** [8:12]
Um, which I, I like that original com, uh, vision. Um, y- your, your messaging around about then was also that you're creating optimized data structures.

**Jerry Liu** [8:22]
Mm-hmm.

**Swyx** [8:22]
Um, d- how f- how, like what's the sort of journey to that, and, like, how does that, uh, contrast with LlamaIndex today?

**Jerry Liu** [8:29]
Yeah. So okay, maybe I can tell a little bit about, like, the beginning intuitions.

**Swyx** [8:34]
Yeah.

**Jerry Liu** [8:34]
Um, I think when I first started, this really wasn't supposed to be something that was, like, a toolkit that people use. It was more just like a system. Um, and the way I wanted to think about the system was more a thought exercise of how language models with their reasoning capabilities, if you just treat them as like brains, can organize information and then traverse it.

**Swyx** [8:52]
Yeah.

**Jerry Liu** [8:52]
So I didn't want to think about embeddings, right? To me, embeddings just felt like it was just an external-

**Swyx** [8:56]
People level

**Jerry Liu** [8:57]
... thing that was like... Well, well it was just external to trying to actually tap into the capabilities of language models themselves, right? I really wanted to see, you know, just as like a human brain could, like, synthesize stuff, could we create some sort of, like, structure where the, this, this, like, neural CPU, if you will, can, like, organize a bunch of information, you know, auto summarize a bunch of stuff, uh, and then also traverse the structure that it created.

That was the inspiration for this initial, like, tree index. Uh, it didn't actually... Like, to, like, to be honest, and I think I said this in the first tweet, it didn't actually work super well, right?

**Swyx** [9:25]
Yeah.

**Jerry Liu** [9:25]
Like, GPT-3 at the time-

**Swyx** [9:27]
You're very honest about that.

**Jerry Liu** [9:28]
Yeah, no, I mean, uh, it was just like, um, GPT-4 obviously is much better at reasoning. Like, I'm one of the first to say, like, you know, you, you shouldn't use anything pre-GPT-4 for anything that requires, like, complex reasoning, um, because it's just gonna be unreliable.

Okay, disregarding stuff like fine-tuning. But it worked okay, but I think it definitely struck a chord with kind of like the, the Twitter crowd, which was just, like, looking for kind of, um, just, like, new ideas at the time.

I guess just, like, thinking about how you can actually bake this into some sort of application because I think what I also ended up discovering was the fact that there, um, basically everybody... There, there was starting to become a wave of developers building on top of GPT-3, and people were starting to realize that what makes them really useful is to apply them on top of your personal data.

And so even if the solution itself was kind of, like, primitive at the time, like, the problem statement itself was very powerful. And so I think being motivated by the problem statement, right, like this broad mission of how do I unlock LLMs on top of the data, also, uh, contributed to the development of LlamaIndex to the state it is today.

And so I think part of the reason, you know, our toolkit has evolved beyond, um, the, like, just existing set of, like, data structures is we really try to take a step back and think, "Okay, what, what exactly are the tools that would actually make this useful for a developer?"

And then, you know, somewhere around December, we made an active effort to basically, like, push towards that direction, make the code base more modular, right, more friendly as an open source library, uh, and then also start adding in, like, embeddings, start thinking into practical, uh, considerations like latency, cost, performance, those types of things.

And then, um, really motivated by that mission, like, start expanding the scope of the toolkit towards, like, covering the, the life cycle of, like, data ingestion and, and querying.

**Swyx** [11:05]
Yeah. Where you also added Llama Hub and, and-

**Jerry Liu** [11:08]
Yeah, yeah. So, so I think that was in, like, uh, January on the data loading side, and so we started adding, like, some data loaders, saw an opportunity there, um, started adding more stuff on the retrieval querying side, right?

We still had, like, the core data structures, but how do, how do you actually make them more modular and kind of like decouple, um, storing state from the types of, like, queries that could run on top of this a little bit?

Um, and then starting to get into more complex interactions like train-of-thought reasoning, routing, and you know, like agent loops.

**Swyx** [11:35]
Yeah, yeah. Very cool.

### Founding

**Alessio** [11:36]
And then you and I spent a bunch of time earlier this year talking about Llama Hub, what that might become. Um, you were still at, at Robust. When did you decide it was time to start the company and then start to think about LlamaIndex is today

**Jerry Liu** [11:53]
Probably December. Yeah. Um, and, and so it was clear that, you know, it was kind of interesting. I was getting some inbound from initial VCs. I was talking about this project, and then in the beginning, I was like: "Oh yeah, you know, this is just like a design project, but, you know, what about my other idea on, like, video data?"

Right? And then I was trying to, like, get, get, um, their thoughts on that. And then everybody was just like: "Uh, yeah, whatever. Like, that part's, like, a crowded market." Uh, and then it became clear that, you know, this was actually a pretty big opportunity, and, uh, like coincidentally, right, like this actually did relate to...

Like, my interests have always been at the intersection of AI, um, data, and kind of like building practical applications, and it was clear that this was evolving into a much bigger opportunity than the previous idea was. Um, so around December, and then I think I, I gave a pretty long notice, but I left, um, officially, like, early March.

**Alessio** [12:36]
What were your thinkings in terms of, like, moats and, uh, you know, founders kind of like overthink it sometimes.

**Jerry Liu** [12:43]
Yeah. Oh, yeah.

**Alessio** [12:43]
You obviously had, like, a lot of open source love and, like, a lot of community, and yeah, like, were you ever thinking, "Okay, I don't know. This is maybe not enough to start a company," or did you always have conviction about it?

**Jerry Liu** [12:56]
Oh, no. I mean, 100%. I felt like I did this exercise, like, um, honestly, probably more, um, late December and then early January, uh, 'cause I was just existentially worried about whether or not this would actually be a company at all.

Um, and okay, what were the key questions I, I was thinking about? And these were the same things that, like, other founders, uh, investors, and also, like, friends would ask me is just like, "Okay, what happens if context windows get much bigger?

Uh, what's the point of actually structuring data, right, in, in the right way?" Um, right. "Why don't you just dump everything into the prompt? Uh, fine-tuning, like what if you just train the model over this data? Um, and then, you know, what's the point of doing this stuff?"

Uh, and then, um, some other ideas is what if, like, OpenAI actually, um, just, like, takes this, uh, like, you know, builds upwards, uh, uh, on top of the, their existing, like, foundation models and starts building in some, like, built-in orchestration capabilities around stuff like RAG and agents and those types of things.

And so I basically ran through this mental exercise and, and, you know, I'm happy to talk a little bit more about those thoughts as well. But at a high level, uh, while context windows have gotten bigger, but there's obviously still a need for, for RAG.

Um, I think RAG is just, like, one of those things that, like, in general, what people care about is, yes, they do, do care about performance, but they also care about stuff like latency and cost. And my entire reasoning at the time was just like: Okay, like, yes, maybe we'll have, like, much bigger context windows as w- as we've seen with, like, 100K context windows.

But for enterprises like, you know, data, which is not in just, like, the scale of, like, a few documents, it's, uh, usually in, like, gigabytes, terabytes, petabytes, like, how do you actually just unlock language models over that data, right?

And so it was clear there was just like, uh, whether it's RAG or some other paradigm, no one really knew what that answer was, and so there was clearly, like, technical opportunity here. Like, there was just, uh, stacks that needed to be invented to actually solve this type of problem, because language models themselves didn't have access to this data.

And so if, like, you just dumped all this data into... Let's say a model had, like, hypothetically an infinite context window, right? And you just dump, like, 50 gigabytes of data into a context window. That just seemed very inefficient to me, because you have these network transfer costs of uploading 50 gigabytes of data to get back a single response.

And so I kind of realized, you know, there's always gonna be some curve regardless of, like, the performance of the best performing models of, like, um, cost versus performance. Um, and so, um, what RAG does is it does provide extra data points along that axis because you kind of control the amount of context you actually want it to retrieve.

Uh, and of course, like, RAG as a term was still evolving back then, but it was just this whole idea of, like, how do you just fetch a bunch of information to actually, you know, like, stuff into the prompt.

And so people even back then were kind of thinking about some of those considerations.

**Alessio** [15:31]
And then you fundraised in June, or, well, you announced your fundraise in June-

**Jerry Liu** [15:34]
Yeah

**Alessio** [15:35]
... um, with Greylock.

**Jerry Liu** [15:36]
Mm-hmm.

**Alessio** [15:37]
Um, how was that process, uh, y- just, like, take us through that process of thinking about the fundraise and, um, your plans for the company, uh, you know, at the time.

**Jerry Liu** [15:46]
Yeah, definitely. I mean, I think we knew we wanted to... I mean, obviously, we knew we wanted to fundraise. I think w- uh, obvious- there was also a bunch of, like, investor interest, and it was probably pretty unusual given the, you know, like, hype wave of generative AI.

So, like, a lot of investors were kind of reaching out around, like, December, January, February. In the end, we went with Greylock. Greylock's great. You know, they've, they've been great partners so far. Um, and, like, to be honest, like, there's, there's a lot of, like, great VCs out there, um, a lot of them who are specialized in, like, open source, data infra, and that type of stuff.

Um, what we really wanted to do was, um, because for us, like, time was of the essence, like, we wanted to ship very quickly and still kind of build mindshare in the space, we just kept the fundraising process very efficient.

I think we basically did it in, like, a week, um, or, or, like, three days.

**Alessio** [16:26]
Excellent.

**Jerry Liu** [16:27]
And so-

**Alessio** [16:27]
Yeah. And-

**Jerry Liu** [16:28]
Just, like, front loaded it, and then, and then just, uh, like-

**Alessio** [16:31]
Then picked the one named Jerry.

**Jerry Liu** [16:33]
Uh, yeah, exactly. Um...

**Alessio** [16:36]
Yeah. I'm, I'm kidding.

**Jerry Liu** [16:37]
Got it.

**Alessio** [16:38]
Uh, he, I mean, he's obviously great, and, and, and Greylock's a fantastic firm.

**Jerry Liu** [16:41]
Yeah, no, and, and, uh, embedding some light research. So, so yeah, just, um, we, we picked, we, we picked Greylock. Uh, they've been great partners. Um, I think in general, when I talk to founders about, like, the fundraise process-

**Alessio** [16:51]
Yeah

**Jerry Liu** [16:52]
... um, it's never, like, the most fun period, I think.

**Alessio** [16:54]
Yeah.

**Jerry Liu** [16:54]
Um, because it's always just, like, you know, uh, there's a lot of logistics. There's lawyers you have to, you know, uh, get in the loop, and then y- and, like, a lot of founders just want to go back to building.

Um, and so I think w- in the end, we're happy that we kept it a pretty efficient process. Yeah.

**Alessio** [17:07]
Cool. Um, and so you fundraised with Simon, uh, your co-founder.

**Jerry Liu** [17:10]
Mm-hmm.

**Alessio** [17:10]
And, um, how do you split things with him? How big is your team now?

**Jerry Liu** [17:13]
The team is growing. Um, by the time this podcast is released-

**Alessio** [17:17]
Mm-hmm

**Jerry Liu** [17:17]
... uh, we'll probably have had one more person join the team. Um, and so basically, uh, it's between, uh, we're rapidly getting to, like, eight or nine people.

**Alessio** [17:25]
Yeah.

**Jerry Liu** [17:25]
Uh, at the current moment, we're around, like, six.

**Alessio** [17:27]
Yeah.

**Jerry Liu** [17:27]
And so just, like, um, there, there'll be some exciting developments in the next few weeks that I'm excited to, to kind of- Um, to, to announce that we've been pretty, um, uh, selective in terms of, like, how we, like, grow the team, obviously.

Like, we look for people that are really active in terms of contributions to LlamaIndex, people that have, like, very strong engineering backgrounds. And primarily, we've been kind of just looking for builders, uh, people that kind of, like, grow the open source and also eventually this, like, managed, like, enterprise platform as well with us.

Um, in terms of, like, uh, Simon, yeah, I've, I've known Simon for a few years now. I knew him back at Uber ATG in Toronto. Um, he's, you know, one of the smartest people I knew. Um, uh, like, you know, has a sense of both, like, a deep understanding of ML, but also just, like, first principles thinking about, like, engineering and technical concepts in general.

And I think one of my criteria is when I was, like, looking for a co-founder for this project, was someone that was, like, technically better than me, 'cause I knew I wanted, like, a CTO. And so honestly, like, there weren't a lot of people that...

I mean, there, there's... I know a lot of people that are smarter than me, but, like, that fit that bill, were willing to do a startup, and also just had the same, like, values that I shared, right? And just, I think doing a startup is very hard work, right?

It's not like... I, I'm sure many, like, you guys all know this. It's, it's a lot of hours, um, a lot of late nights, um, and you want to be, like, in the same place together and just, like, being willing to hash out stuff and, and have that grit, basically, and I, I really look for that.

And so Simon really, um, fit that bill, and I think I convinced him to bring- uh, jump on board.

**Alessio** [18:45]
Yeah. Yeah. Nice job. Um, and obviously I've had the pleasure of chatting and working with, uh, a little bit with both of you. Um, what would you say those, those, like, your top, like, one or two values are when, when thinking about that?

Or the sh- culture of the company and that kind of stuff.

**Jerry Liu** [18:58]
Yeah. Well, I, I think in terms of, um, the culture of the company, it's, it's really, like, uh... I mean, there's a few things I can name off the top of my head. Uh, one is just, like, uh, passion, integrity.

I think that's very important for us. We wanna be honest. We don't wanna, like, obviously, like, copy code or, or kind of like, you know, just like, you know, not give attribution, those types of things, and, and just, like, be true to ourselves.

I think we're all very, like, down to earth, like, humble people. But obviously, I think just willingness to just, like, own stuff and dive right in, and I think grit comes with that. I, I think in the end, like, this is a very fast-moving space, and we want to just, like, be one of the, you know, like, dominant forces in helping to provide, like, production quality LLM applications.

**Alessio** [19:36]
Yeah. Um, so I, I promise we'll get to more technical questions soon.

**Jerry Liu** [19:39]
No, no worries.

**Alessio** [19:40]
But I also want to impress on the audience, um, that this is a, you know, very conscious and, um, intentional company building. And since your, um, fundraising post, which was in June, um, and now it's, it's September, so it's been about three months, you've actually gained 50%, uh, in, in terms of stars and followers.

You've 3X'd your download count to 600,000 a month, uh, and your Discord m-membership has reached 10,000. So, like, a lot of c- ongoing growth.

**Jerry Liu** [20:06]
Yeah, definitely. And, and obviously there's a lot of room to expand there too.

**Alessio** [20:10]
Yeah.

**Jerry Liu** [20:10]
Um, and so open source growth is gonna continue to be one of our core, uh, goals. Uh, 'cause in the end, it's just like, we want this thing to be, well, one, big, right? We all have, like, uh, big ambitions.

### Context & RAG

**Jerry Liu** [20:19]
But two, just, like, really provide value to developers in helping them in prototyping and also productionization of their apps. Um, and I think it turns out we're in the fortunate circumstance where a lot of different companies and individuals, right, are in that phase of, like, you know, maybe they've hacked around on some initial LLM applications, but they're also looking to, you know, start to think about what are the production grade challenges necessary to actually, um, you know, uh, uh, that to solve to actually make this thing robust and reliable in the real world.

And, and so we want to basically provide the tooling to do that, and to do that, we need to both spread awareness and education of a lot of the key practices of what's going on. And so a lot of this is going to be continued growth, expansion, and education, and we, we do prioritize that very heavily.

**Alessio** [21:00]
Awesome. Um, let's dive into some of the questions you were asking yourself, uh, initially around fine-tuning and RAG, how these things play together.

**Jerry Liu** [21:09]
Yep.

**Alessio** [21:10]
Um, you mentioned context. What is the minimum viable context for RAG? So what's, like, a context window too small? And at the same time, maybe what's, like, a maximum context window? We talked before about the LLMs are U-shaped reasoners, so as the context got larger, like, it really only focuses on the end and the start of the prompt, and then it kinda, um, peters down.

Any learnings, any kind of like tips you wanna give people, um, as they think about it?

**Jerry Liu** [21:40]
So this is a great question, and, um, I think part of what I wanted to, um, kind of like talk about at conceptual level, especially with the idea of, like, thinking about what is a minimum context. Like, okay, what if the minimum context was, like, 10 tokens versus, like, you know, 2K tokens versus, like, a million tokens, right?

Like, and what does that really give you, and what are the limitations if it's, like, 10 tokens? It's kind of like, um, like 8-bit, 16-bit games-

**Alessio** [22:02]
Mm

**Jerry Liu** [22:02]
... right? Like back in the day, like if you play Mario-

**Alessio** [22:05]
Mm

**Jerry Liu** [22:05]
... um, and you have, like, the initial Mario where the graphics were very blocky and, and now obviously it's like full HD 3D. Just the resolution of the context and the output will change, uh, depending on how much context you can actually fit in.

Um, the, the way I kind of think about this from a more principled manner is, like, there's this concept of, like, um, information capacity, um, just this idea of, like, entropy. Like, given any fixed amount of, like, storage space, like, how much information can you actually compact in there?

And so basically, a context window length is just, like, some fixed amount of storage space, right? And so there's some theoretical limit to the maximum amount of information you can compact into, like, a 4,000 token storage space. And what is that storage space used for these days with LLMs?

It's for inputs and also outputs. And so this really controls the maximum amount of information you can feed in terms of the prompt plus the granularity-

**Alessio** [22:52]
Mm-hmm

**Jerry Liu** [22:52]
... of output. If you had an infinite context window, you could have an infinitely detailed response-

**Alessio** [22:55]
Yeah

**Jerry Liu** [22:55]
... and also infinitely detailed memory. But if you don't, you can only kind of represent stuff in more quantized bits, right? And so the smaller the context window, uh, just, uh, generally speaking, the less details and, and maybe the less, um, infor- like specific, precise information you're gonna be able to surface at any given point in time.

**Alessio** [23:12]
And when you have short context, is the answer just, like, get a better model? Or is the answer maybe, "Hey, there needs to be a balance between fine-tuning and RAG to make sure you're gonna, like, leverage the context, but at the same time, don't keep it too low resolution"?

**Jerry Liu** [23:27]
Yeah, yeah. Well, there's probably some minimum thr- like, I don't think anyone wants to work with, like, a 10... I mean, that's just a thought exercise anyways, a 10 token context window. Um, I think nowadays the modern context window is, like, 2K, 4K is enough for, for just, like, doing some sort of retrieval on granular context and being able to synthesize information.

I think for most intents and purposes, that level of resolution is probably fine for most people for most use cases. I think the limitation's actually more on, okay, if you're gonna actually combine this thing with some sort of retrieval data structure mechanism, there's just limitations on the retrieval side, um, because maybe you're not actually fetching the most relevant context to actually answer this question, right?

Like yes, y- like given the right context, 4,000 tokens is enough, but if you're just doing like top case similarity, like you might not be, be-

**Alessio** [24:09]
Mm-hmm

**Jerry Liu** [24:09]
... fetching the right information from the documents.

**Alessio** [24:11]
Yeah. So how should people think about when to stick with RAG versus when to even entertain fine-tuning? And also in terms of what's like the threshold of data that you need to actually worry about fine-tuning versus like just stick with RAG?

Obviously, you're biased because you're building a RAG company, but, uh ...

### RAG vs Fine-tuning

**Jerry Liu** [24:29]
No, no, actually, um, I think I have like a few hot takes in here, some of which sound like a little bit contradictory-

**Alessio** [24:34]
Mm-hmm

**Jerry Liu** [24:35]
... to what we're actually building. To be honest, I don't think anyone knows the right answer. Uh, I think this is just something-

**Alessio** [24:38]
We're pursuing the truth.

**Jerry Liu** [24:39]
Yeah, exactly. This is just like thought exercise towards like understanding the truth, right? So, so I think, um, okay, I have a few hot takes. One is like RAG is basically just, just a hack that turns out it's a very good hack.

Um, because, uh, what is RAG? RAG is you keep the model fixed and you just figure out a good way to like stuff stuff into the prompt of, of the language model. Everything that we're doing nowadays in terms of like stuffing stuff into the prompt is just algorithmic.

We're just figuring out nice algorithms to, to like retrieve right information-

**Alessio** [25:06]
Mm-hmm

**Jerry Liu** [25:06]
... with top case similarity, do some sort of like hybrid search, some sort of like a train of thought decomp, and then, and it just like stuffs stuff into a prompt. So it's all like algorithm- algorithmic. Um, and it's more like just software engineering to try to make the most out of these like existing APIs.

The reason I say it's a hack is just like from a pure like optimization standpoint, if you think about this from like the machine learning lens and less the software engineering lens, there's pieces in here that are gonna be like suboptimal, right?

Like obviously, like the thing about machine learning is when you optimize like some system that can be optimized within machine learning, like the set of parameters, you're really like changing like the entire system's weights to try to optimize this objective function.

And if you just cobble a bunch of stuff together, you can't really optimize the pieces that are inefficient, right? And so like a retrieval interface, like doing top K embedding lookup, that part is, uh, inefficient because there might be potentially a better, more learned retrieval algorithm that's better.

Um, if, you know, you, um, kinda do stuff like some sort of... I know nowadays there's this concept of how do you do like short-term or long-term memory, right? Like represent stuff in some sort of vector embedding, do trunk sizes, all that stuff.

It's all just like decisions that you make that aren't really optimized, right? It's, it's more... And it's not really automatically learned, it's more just things that you set beforehand to actually feed into the system. There's a lot of room to actually optimize the performance of an entire LLM system, um, potentially in a more like machine learning based way, right?

And, and I, I will leave room for that. And this is also why I think like in the long term, like I do think fine-tuning will probably have like greater, um, importance. Um, and, and, and just like there will probably be new architectures invented that where you can actually kind of like include a lot of this under the black box as opposed to having like cobbling, uh, together a bunch of components outside the black box.

That said, just pra- very practically, given the current state of things, like even if I said RAG is a hack, it's a very good hack and it's also very easy to use, right? And so just like for kind of like the AI engineer per- persona that like, uh, which, uh, to be fair is kind of one of the reasons generative AI has gotten so big is because it's way more accessible for everybody to get into as opposed to just like traditional machine learning.

Um, it tends to be good enough, right? And if we can basically provide these existing techniques to help people really optimize how to use existing systems without having to really deeply understand machine learning, I still think that's a huge value add.

And so there's, uh, very much like a UX and ease of use problem here, which, which is just like RAG is way easier to onboard and use. Um, and that's probably like the primary reason why everyone should do RAG instead of fine-tune it to begin with.

Um, if you think about like the 80/20 rule, like RAG very much fits within that and fine-tuning it doesn't really right now. Um, and then I'm just kind of like leaving room for the future that, you know, like in the end, um, fine-tuning can probably take over some of the aspects of, of like what RAG does.

**Alessio** [27:51]
I don't know if this is mentioned in your, uh, in your recap there, but, um, explainability also allows for sourcing and, and like at, at the end of the day, like to increase trust, we have to source documents.

**Jerry Liu** [28:04]
Yeah. So, so I think what RAG does is it increases like transparency, visibility into the actual documents-

**Alessio** [28:09]
Right

**Jerry Liu** [28:09]
... that are getting fed into the context.

**Alessio** [28:10]
It's like directly, here's where they got it from.

**Jerry Liu** [28:12]
Exactly. Um, and so that's definitely an advantage. I think the other piece that I think is an advantage, and I think this is something that, um, someone actually brought up, is just, uh, you can do access, access control with, with RAG if you have-

**Alessio** [28:22]
Yeah

**Jerry Liu** [28:22]
... an external storage system. Um, you can't really do that with, with large language models. It's just like gate information to the neural net weights like, um, depending on the type of user.

**Alessio** [28:31]
Mm-hmm.

**Jerry Liu** [28:31]
For the first point, you could technically, right, you could technically have the language model, like if it memorized e- enough information just to like cite sources, but there's a question of just trust whether or not you actually-

**Alessio** [28:44]
It's gonna make it up. Yeah.

**Jerry Liu** [28:44]
Yeah. Well, but, but like it, it makes it up right now 'cause it's like not good enough. But imagine a world where it is good enough and it does give accurate citations.

**Alessio** [28:51]
Yeah.

**Jerry Liu** [28:51]
Yeah.

**Alessio** [28:52]
Yeah. No, I, I think to establish trust you just need a direct, uh, connection. So it's, it's kind of weird. It's, it's, it's this melding of, you know, deep learning systems versus tr- very traditional, uh, information retrieval .

**Jerry Liu** [29:04]
Yeah, exactly. Well, so, so I think, I mean, I kinda think about it as analogous to like humans, right? Like, uh, we as humans, obviously we use the internet, we use tools.

**Alessio** [29:12]
Yeah.

**Jerry Liu** [29:12]
Uh, these tools have API interfaces are well defined.

**Alessio** [29:14]
Yeah.

**Jerry Liu** [29:14]
Um, and obviously we're not, like the tools aren't part of us and so we're not like back propping or optimizing over these tools. Uh, and so kind of when you think about like RAG, it's basically, um, uh, LLM is learning how to use like a vector database to look up information that it doesn't know.

And so then there's just a question of like how much information is inherent within the network itself and how much does it need to do some sort of like tool use to look up stuff that it doesn't know.

And I do think there will probably be more and more of that interplay as time goes on.

**Alessio** [29:41]
Yeah. Um, some follow-ups on the

**Swyx** [29:44]
... discussions that we've had. Uh, so I, uh, um, you know, we discussed fine-tuning a bit, and what's your current take on whether you can, you can fine-tune new knowledge into LLMs?

**Jerry Liu** [29:53]
That's one of those things where I think long term you definitely can. Um, I, I think some people say you can't. I, I disagree. I think you-

**Swyx** [29:58]
Yep

**Jerry Liu** [29:58]
... definitely can. Just right now I haven't gotten it to work yet. So, so I think, like, with-

**Swyx** [30:01]
You've tried.

**Jerry Liu** [30:02]
Yeah, well, um, not in a very principled way, right? Like, this is something that requires, like, an actual research scientist and not someone that has, like, you know, an hour or two per night to actually look into-

**Swyx** [30:10]
I mean, you, like-

**Jerry Liu** [30:10]
I, I-

**Swyx** [30:10]
You were a research scientist at Uber, I mean.

**Jerry Liu** [30:12]
Yeah, but, but it's, like, full-time, full-time looking into it. So, so I think, um, what I specifically concretely did was I took OpenAI's fine-tuning endpoints and then tried to... You know, it's in, like, a chat message interface, and so there's, like, a user assistant message format.

And so what I did was I tried to take just some piece of text and have the LLM memorize it by just asking it a bunch of questions about the text. So given a bunch of context, I would generate some questions and then generate some response and just fine-tune over the question responses.

Um, that hasn't really worked, uh, super well. Um, but that's also because I'm, I'm just, like, trying to, like, use OpenAI's endpoints as is. Um, if you just think about, like, traditional, like, how you train a transformers model, um, there's kind of like the, uh, instruction and, like, fine-tuning aspect, right?

You, like, um, kind of ask it stuff and guide it with correct responses, but then there's also just, like, um, next token prediction. Um, and that's something that you can't really do with the OpenAI API, but you can do with if you just train it yourself.

Um, and that's probably possible if you just, like, train it over some corpus of data. I think Shashua from Berkeley said, like, you know, when they trained Gorilla, they were like, "Oh, you know, this, uh, a lot of these LLMs are actually pretty good at memorizing information."

Um, just the way the API interface is exposed is just no one knows how to use them right now, right? And so, so I think that's probably one of the issues.

**Swyx** [31:23]
Uh, just to clue people in who haven't read the paper, Gorilla is the one where they, uh, trained to, uh, use specific APIs?

**Jerry Liu** [31:30]
Yeah, yeah, and I think they also did something where, like, the model itself, um, could learn to, um... I for- Yeah, I think this was in the Gorilla paper. Like, the, the, the model itself could, uh, try to learn some prior over the data to decide, like, what tool to pick, but there's also...

It's also augmented with retrieval that helps supplement it in case, like, the, the, the, um, prior doesn't actually work.

**Swyx** [31:51]
Mm-hmm. Is that something that you'd be interested in supporting?

**Jerry Liu** [31:55]
I mean, I think in the long term, like, if, like, this is kind of how fine-tuning, like, RAG evolves, like, I, I do think there will be some aspect where fine-tuning will probably memorize some high-level concepts of knowledge, but then, like, RAG will just be there to supplement, like, aspects that it doesn't know.

Yeah.

**Swyx** [32:09]
Yeah, yeah. Awesome.

**Jerry Liu** [32:10]
Um, obviously RAG is the default way. Like, to be clear, RAG right now is the default way to actually augment stuff with knowledge. Um, I think it's just an open question of how much the LLM can actually internalize both high-level concepts but also details, um, uh, as you can, like, train stuff over it.

And coming from a ML background, like, there is a certain beauty in just baking everything into some training process of a, of a language model. Like, um, if you just take raw ChatGPT or ChatGPT Code Interpreter, right, like GPT-4, it's not like you do RAG with it.

You just ask it questions about, like, "Hey, how do I, like, define a Pydantic model in Python?"

**Swyx** [32:44]
Yeah, that's great.

**Jerry Liu** [32:44]
And then, like, "Can you give me an example? Can you visualize a graph for me?" It just does it, right?

**Swyx** [32:47]
Yeah.

**Jerry Liu** [32:48]
Like, and, and it'll run it through Code Interpreter as a tool, but that's not, like, a source for knowledge. That's just an execution environment. And so there is some beauty in just, like, having the model itself, like, just, you know, instead of you kind of defining the algorithm for what the data structure should look like, the model just learns it under the hood.

Um, that said, I think the reason it's not a thing right now is just, like, that no one knows how to do it. It probably costs too much money, and then also, um, like, the API interfaces and, and just, like, the, the actual, like, uh, ability to kind of evaluate and improve on performance, um, like, isn't known to most people.

**Swyx** [33:19]
Yeah. Um, it also would be better with browsing.

**Jerry Liu** [33:24]
Yeah.

**Swyx** [33:24]
Uh, I wonder when they're gonna put that back.

**Jerry Liu** [33:26]
Okay.

**Swyx** [33:28]
Um, okay, cool. Um, yeah, so and then one more follow-up, uh, before we go into RAG for AI engineers, uh, is on your men- your brief mention about, um, security or auth.

**Jerry Liu** [33:37]
Mm-hmm.

**Swyx** [33:37]
Um, and how many of your, the people that you talk to, you know, you talk to a lot of, uh, people putting LlamaIndex into, uh, production. How many people actually are there with it for versus just, like, let's just dump a whole company Notion into this thing?

**Jerry Liu** [33:50]
Wait, are we talking about from, like, the security auth standpoint?

**Swyx** [33:52]
Uh, yeah, how many-

**Jerry Liu** [33:53]
Oh, I see

**Swyx** [33:53]
... how, how, how big a need is that? Because I, I, um, I talked to some people who are thinking about building tools in that domain, but I don't know if people want it.

**Jerry Liu** [34:01]
I mean, I think bigger companies, like, um, just bigger companies like banks, consulting firms, like, they all want this.

**Swyx** [34:08]
Yeah, it's a requirement, right?

**Jerry Liu** [34:09]
Um, the way they're using LlamaIndex is not, uh, with this obviously, 'cause I don't think we have support for, like, access control or auth or that type of stuff, like, under the hood, 'cause we're more just like an orchestration framework.

**Swyx** [34:18]
Mm-hmm.

**Jerry Liu** [34:18]
Um, and so the way they do it, they, they build these initial apps is more kind of like prototype, like, let's kind of, yeah, like, you know, use some publicly available data that's not super sensitive. Let's, like, you know, assume that every user is gonna be able to have access to the same amount of knowledge, those types of things.

Um, I think users have asked for it, but I don't think that's, like, a P0. Like, I think the P0-

**Swyx** [34:36]
Mm

**Jerry Liu** [34:36]
... is more on, like, can we get this thing working-

**Swyx** [34:38]
What exists today

**Jerry Liu** [34:38]
... before we expand this to, like, more users within the org?

**Swyx** [34:41]
Yep. Cool.

**Alessio** [34:44]
So there's a bunch of pieces to RAG, obviously. It's not a, just a acronym.

### Inside LlamaIndex

**Jerry Liu** [34:48]
Mm-hmm.

**Alessio** [34:48]
And you tweeted recently you think every AI engineer should build it from scratch at least once. Um, why is that?

**Jerry Liu** [34:56]
I think so. Um, I, I, I'm actually kind of curious to hear your thoughts about this. Um, but this kind of relates to the initial, like, AI engineering posts that you put out. Um, and then also just, like, the role of an AI engineer and the skills that they're gonna have to learn to truly succeed.

Um, 'cause there's an entire spectrum. On one end, you have people that don't really, uh, like, understand the fundamentals and just wanna use this to, like, cobble something together to, to build something. And I think there is a beauty in that, for what it's worth.

Like, it's just one of those things. And, and Gen, GenAI has made it so that you can just use these models in inference-only mode, cobble something together, use it to power your app experiences. On the other end, what we're increasingly seeing is that, like, more and more developers building with these apps start running into honestly, like, pretty similar issues that, like, will plague just a standard ML engineer building, like, a classifier model, which is just, like, accuracy problems, like, and hallucinations basically are just an accuracy problem, right?

Like, it's not giving you the right results, so what do you do? You have to iterate on the model itself. You have to figure out what parameters you tweak. You have to gain some intuition about this entire process.

That workflow is pretty similar, honestly, like even if you're not training the model, to just like tuning a ML model with like hyperparameters and learning like proper ML practices of like, okay, how do I have, like define a good evaluation benchmark?

**Swyx** [36:07]
Mm.

**Jerry Liu** [36:07]
How do I define like the right set of metrics to use, right? How do I actually iterate and improve the performance of this pipeline for production? What tools do I use, right? Like every ML engineer uses like some form of Weights & Biases TensorBoard or like some other experimentation tracking tool.

Like, um, what tool should I use to actually help build like LLM applications and optimize it for production? There's like a certain amount of just like LLM ops, like tooling and concepts and, and just like practices that people kind of have to internalize if they want to optimize these.

And so I think the, the reason I, I think like being able to build like RAG from scratch is important is it really gives you a sense of like how things are working to g- help you build intuition about like what parameters there are within a RAG system and which ones actually tweak to make them better.

One of the advantages of LlamaIndex, the LlamaIndex quick start is it's three lines of code. Um, the downside of that is you have zero visibility into what's actually going on under the hood, and I think this is something that we've kind of been thinking about for a while, and I'm like, "Okay, let's just release like a new tutorial series that's just like we- instead, not, no three lines of code.

We're just gonna go in and actually show you how the thing actually works-

**Swyx** [37:08]
Yeah

**Jerry Liu** [37:08]
... under the hood," right?

**Swyx** [37:08]
Yeah.

**Jerry Liu** [37:08]
And so I... Like, does everybody need this? Like, probably not. Like as for some people, the three lines of code might work. Um, but I think increasingly, like, like honestly, 90% of the users I talk to have questions about how to improve the performance of their app, and so just like given this is just like one of those things that's like better for the understanding.

**Swyx** [37:25]
Yep. I'd say, uh, it is one of the most useful tools of any sort of developer education toolkit to w- write things yourself from scratch. Um, so Kelsey Hightower famously wrote, uh, "Kubernetes the Hard Way," which is don't use Kubernetes-

**Jerry Liu** [37:41]
Oh.

**Swyx** [37:41]
... just like do everything. Here's h- here's everything that you would have to do, uh, by yourself, and you should be able to put all these things together yourself to understand the value of Kubernetes.

**Jerry Liu** [37:49]
Mm-hmm.

**Swyx** [37:50]
Um, and the same thing for LlamaIndex. Uh, I've done... Uh, I was the guy who did the same for React.

**Jerry Liu** [37:54]
Mm.

**Swyx** [37:54]
Um, and yeah, it's pretty... Well, it's pretty, it's pretty good exercise for you to just fully understand everything that's going on under the hood. Um, and I was, I was actually gonna suggest g- well, in, in one of the previous conversations, um, you know, there's all these like hyperparameters, like the, the-

**Jerry Liu** [38:08]
Mm

**Swyx** [38:08]
... size of the chunks and all that. Um, and, uh, I was, I was thinking like, you know, what, what if, what if, what, what, what, what would hyperparameter opti- optimization for RAG look like?

**Jerry Liu** [38:18]
Yeah, definitely.

**Swyx** [38:19]
That might be interesting.

**Jerry Liu** [38:19]
I mean, so absolutely. I think that's gonna be a increasing thing. I think that's something worth kind of looking at 'cause like-

**Swyx** [38:24]
I think someone should just put, do like some large scale study and then just ablate everything and just... You, you tell us.

**Jerry Liu** [38:31]
I, I think it's gonna be hard to find a universal default that works for everybody.

**Swyx** [38:34]
Oh.

**Jerry Liu** [38:35]
I, I think it's gonna be somewhat-

**Swyx** [38:35]
Are you telling me to boo?

**Jerry Liu** [38:40]
I, I do think it's gonna be somewhat-

**Swyx** [38:41]
Yeah, yeah

**Jerry Liu** [38:41]
... like dependent on the data and use case. I, I think if there was a universal default, that'd be amazing. Um, but I, I think for increasingly we found, you know, people are just defining their own like custom parsers for like PDFs, Markdown files for like, you know, SEC filings versus like, you know, Slack conversations.

Uh, and then like the use case too, like do you want like a summarization, like the granularity of the response, like it really affects the parameters that you want to pick. Um, and so I, I do like the idea of hyperparameter optimization though, but it's kind of like one of those things where you are kind of like training the model basically, um, kind of on your own data domain.

**Swyx** [39:13]
Yeah. Uh, you mentioned custom parsers. Um, you've designed LlamaIndex, so maybe we can talk about like the surface area of-

**Jerry Liu** [39:18]
Mm

**Swyx** [39:18]
... the framework. Uh, you designed LlamaIndex in a way that it's more modular, yeah, like, like you mentioned. Um, how would you describe the different components and what's customizable, uh, in each?

**Jerry Liu** [39:29]
Yeah, I think they're all customizable. Um, and I think that there, there is a certain burden on us to make that more clear through the docs. Um, but-

**Swyx** [39:36]
Well, number four is customization tutorials, so.

**Jerry Liu** [39:38]
Yeah, yeah, but, but I think like just in, in general, I think we, we do try to make it so that you can plug in the out-of-the-box stuff, but like if you, if you want to kind of, um, uh, customize more lower level components, like we definitely encourage you to do that and plug it into the rest of our abstractions.

So let me just walk through like maybe some of the basic components of LlamaIndex. There's data loaders. You can load data from different data sources. We have LlamaHub, which you guys brought up, which is, um, you know, a collection of different data loaders of like unstructured, unstructured data, um, like PDFs, file types, like, uh, Slack, uh, Notion, uh, all that stuff.

Um, now you load in this data, we have a bunch of like parsers and, and transformers. You can split the text, you can add metadata to the text, uh, and then basically figure out a way to, uh, load it into like a vector store.

Um, so I mean, you, you worked at like Airbyte, right? It's kind of like there is some aspect like E and T, right?

**Swyx** [40:23]
Mm-hmm.

**Jerry Liu** [40:24]
In, in terms of like transforming this data. Um, and then the L, right, loading it into some storage abstraction, we have like a bunch of integrations with different document storage systems. Um, so that's data. Uh, and then the second piece really is about like, um, how do you retrieve this data?

Um, how do you like synthesize this data? And how do you like do some sort of higher level reasoning over this data? So retrieval is one of the core abstractions that we have. Uh, we do encourage people to like customize, define their own retrievers.

Uh, that's why we have that section on kind of like how do you define your own like custom retriever.

**Swyx** [40:53]
Mm.

**Jerry Liu** [40:53]
But also we have like out-of-the-box ones. Um, the retrieval algorithm kind of depends on how you structure the data, obviously. Like if you just flat index everything with, uh, like chunks with like embeddings, then you can really only do like top K like lookup, uh, plus maybe like keyword search or something.

But if you can index it in some sort of like hierarchy, like define relationships, you can do more interesting things, like actually traverse relationships between nodes. Um, then after you have this data, how do you like synthesize the data, right?

Um, and, and this is the part where you feed it into the language model. There's some response abstraction that can abstract away over like long contexts to actually still give you a response, even if the context overflows the context window.

Uh, and then there's kind of these like higher level like reasoning primitives that I'm gonna define broadly, and I'm just gonna call them in some general bucket of like agents, um, even though everybody has different definitions of agents.

And agents-

**Swyx** [41:43]
But you're the first to data agents, which I was very excited about.

**Jerry Liu** [41:45]
Yeah, we, we kind of like coined, coined that term.

**Swyx** [41:47]
Yeah.

**Jerry Liu** [41:47]
And the way we, we thought about it was, you know, we wanted to think about how to use agents for s- uh, like data workflows basically, and, and so what are the reasoning primitives that you want to do?

So the most simple reasoning primitive you can do is some sort of routing module. Like you can just, um, it's a classifier. Like given a cl- a query, just make some automated decision on what choice to pick, right?

Um, you could use LLMs. You don't have to use LLMs. You could just train a classifier basically. Um, that's something that we, we might actually explore. Uh, and then the, the next piece is, okay, what are some higher level things?

You can have the LLM like define like a query plan, um, right, to actually execute over the data. You can do some sort of while loop, right? That's basically what an agent loop is, which is like react, um, tree of thoughts, um, like train of thought, like the OpenAI function calling like while loop, um, to try to like take a question and try to break it down into some, some, uh, series of steps to actually try to execute to get back a response.

Um, and so there's a range in complexity from like simple reasoning primitives to more advanced ones, and I think that's the way, the way we kind of think about it is like, uh, which ones should we implement and how do they work well?

Like do they work well over like the types of like data tasks that we give them?

**Swyx** [42:50]
How do you think about optimizing each piece? So take, um, embedding models as one piece of it. Uh, you offer fine-tuning embedding models, and I saw it was like fine-tuning gives you like 5, 10% increase.

**Alessio** [43:03]
What's kind of like the delta left on the embedding side? Do you think we can get models that are, like, a lot better? Do you think, like, that's one piece where people should really not spend too much time?

### Optimizing RAG

**Jerry Liu** [43:13]
I mean, I think they should. I, I just think it's, it's not the only parameter, 'cause I think in the end, um, uh, if you think about everything that goes into retrieval, the chunking algorithm, um, how you define, like, uh, metadata, right?

Uh, will bias your embedding representations. Then there's the actual embedding model itself, which is something that you can try optimizing, and then there's, like, the retrieval algorithm. Are you gonna just do top K or are you gonna do, like, hybrid search or are you gonna do auto retrieval?

Like, there's a bunch of parameters. And so I do think it's something everybody should try. Um, I, I think, uh, by default we use, like, OpenAI's embedding model. A lot of people these days use, like, sentence transformers because it's, it's just, like, free, open source, and you can actually optimize, directly optimize it.

Um, this is an active area of exploration. I do think one of our goals is, um, it should ideally be relatively free for every developer to just run some fine-tuning process over their data to squeeze out some more points and performance.

And if it's that relatively free and there's no downsides, everybody should basically do it. Um, there's just some complexities, right, in terms of optimizing your embedding model, especially in a production-grade data pipeline. If you actually fine-tune the embedding model, um, and the embedding space changes, you're gonna have to re-index all your documents.

Uh, and for a lot of people, that's not feasible. Um, and so I think, like, Joe from Vespa on w- our webinar is like, there's this idea that, um, uh, depending on kind of like if you're just using, like, um, document and query embeddings, you could keep the document embeddings frozen and just train a linear transform on the query or, or any sort of transform on the query, right?

So therefore it's just a query side transformation instead of actually having to re-index all the document embeddings. Um, the other piece is-

**Alessio** [44:45]
Wow, that's pretty smart.

**Jerry Liu** [44:46]
Yeah, yeah. So, so I think, uh, we, we, we weren't able to get, like, huge performance gains there, but it does, like, improve performance a little bit, and that's something that basically, you know, everybody should be able to kick off.

You can actually do that on LlamaIndex too.

**Alessio** [44:57]
OpenAI has a cookbook on, uh, adding bias to the embeddings too, right?

**Jerry Liu** [45:01]
Yeah. Yeah, I think so. Yeah, there's just, like, different parameters that you can, you can try adding to try to, like, optimize the retrieval process. Um, and the idea is just like, okay, um, by default you have all this text.

Uh, it kind of lives in some, uh, latent space, right?

**Alessio** [45:15]
Mm-hmm.

**Jerry Liu** [45:15]
Uh-huh.

**Alessio** [45:15]
Yay.

**Jerry Liu** [45:16]
Shout out-

**Alessio** [45:16]
Hey

**Jerry Liu** [45:16]
... shout out latent space. Um-

**Alessio** [45:17]
You should take a drink every time-

**Jerry Liu** [45:19]
Yeah.

**Alessio** [45:20]
... that happens.

**Jerry Liu** [45:20]
But it lives in some latent space. Um, but, like, depending on the type, specific types of questions that the user might wanna ask, the latent space might not be optimized, right? For, for actual, uh, like, to actually, um, retrieve the relevant piece of context that the user wanna ask.

So can you shift the embedding points a little bit, right? And how do we do that, basically? That's really the key question here. So optimizing the embedding model, um, even changing the way you, like, chunk things, these all shift the embeddings.

**Alessio** [45:45]
So the retrieval is interesting. I got a bunch of startup pitches that are like, "Look, RAG is cool, but, like, there's a l- a lot of stuff in terms of ranking that could be better. There's a lot of stuff in terms of, um, uh, sunsetting data once it starts to become stale that could be better."

Um, are you gonna move into that part too? So like, you have SEC Insights as one of kind of like your demos, and that's, like, a great example of, "Hey, I don't wanna embed all the historical documents because a lot of them are outdated, and I don't want them to be in the context."

What's that problem space like? How much of it are you gonna also help with and versus how much you expect others to take care of?

**Jerry Liu** [46:23]
Yeah. I'm, I'm happy to talk about SEC Insights in just a bit. I think more broadly about the, like, overall retrieval space, we're very interested in it because a lot of these are very practical problems that people have asked us.

Um, so the idea of outdated data, I think, um, how do you, like, deprecate or time weight data, um, and do that in a reliable manner, I guess, so you don't just, like, kind of set some parameter and all of a sudden that affects your, all your retrieval algorithms is pretty important because, uh, people have started bringing that up.

Like, "I have a bunch of duplicate documents. Things get out of date. How do I, like, sunset documents?" Um, and then ranking, right? Yeah, so I think this space is not new. Um, I think, uh, like, rather than inventing, like, new retriever techniques for the sake of, like, just inventing better ranking, um, we wanna take existing ranking techniques and kind of, like, package it in a way that's, like, intuitive and easy for people to understand.

That said, I think there are interesting and new retrieval techniques that, uh, are kind of in place that can be done, um, with when you tie it into some downstream RAG system. I mean, like, the, the reason for this is just, like, if you think about how, um, like, the idea of, like, chunking text, right?

Like, that really, that just really wasn't a thing, um, or at least for this specific purpose of, like... Like, the reason chunking is a thing in RAG right now is because, like, you wanna fit within the context window of an LLM, right?

Uh, like, why do you wanna chunk a document? That, that just was less of a thing, I think, back then. If you wanted to, like, uh, transform a document, it was more for, like, structured data extraction or something in the past.

Um, and so there's kind of like certain new concepts that you gotta play with, um, that you can use to, uh, invent kind of more interesting retrieval techniques. Another example here is actually LLM-based reasoning. Like, LLM-based chain of thought reasoning.

You can take a question, break it down into smaller components, and use that to actually, uh, send to your retrieval system, and that gives you better results than just kind of like sending the full question to a retrieval system.

That also wasn't really a thing back then, but then you can kind of figure out an interesting way of, like, blending old and the new, right, with LLMs and, and data.

**Alessio** [48:16]
Yeah. There's a lot of ideas that you come across. Um, do you have a store of them?

**Jerry Liu** [48:23]
So, okay, I, I think the, the, um, uh, yeah, I think I sometimes I get, like, inspiration. There's, like, some problem statement and I'm just like, "Oh, let's, like, hack this out."

**Alessio** [48:30]
Following you, following you is very hard-

**Jerry Liu** [48:31]
Yeah.

**Alessio** [48:31]
... 'cause it's just a, a lot of homework.

**Jerry Liu** [48:33]
So I, I think I, I've, I've started to, um, uh, like, step on the brakes just a little bit 'cause then I, I start-

**Alessio** [48:39]
No, no, no. Keep going, keep going.

**Jerry Liu** [48:40]
No, no, no. Well, the re- the reason is just like, okay, if I just have invent like 100 more retrieval techniques, like, like, sure. But, like, how do people know which one is good-

**Alessio** [48:47]
Yeah

**Jerry Liu** [48:47]
... and which one's, like, bad, right? And so-

**Alessio** [48:48]
Have a librarian, right? Like, it's gonna catalog it and go like-

**Jerry Liu** [48:51]
You're gonna need some, like, benchmarks, and so I think that's probably the, the focus for the next, the next few weeks-

**Alessio** [48:56]
Okay

**Jerry Liu** [48:56]
... is actually, like, properly kind of like having an understanding of, like, oh, you know, when should you do this? Or like wh- does this actually work well?

**Alessio** [49:01]
Yeah. Some kind of like a, maybe like a flowchart-

**Jerry Liu** [49:04]
Mm-hmm

**Alessio** [49:04]
... uh, decision tree type of thing.

**Jerry Liu** [49:05]
Yeah, exactly.

**Alessio** [49:06]
When this, do that. Yeah, something like that. That'd be really helpful for me. Thank you.

**Jerry Liu** [49:11]
Yeah.

### Production

**Alessio** [49:11]
Um, do you wanna talk about SEC Insights?

**Jerry Liu** [49:14]
Um, sure. Yeah

**Swyx** [49:16]
You had a question?

**Alessio** [49:17]
Yeah, yeah. Just, uh, I mean, that's kind of like a good-

**Swyx** [49:20]
It, it seems like your most successful side project.

**Jerry Liu** [49:23]
Yeah. Okay, so what, what is SEC Insights for our listeners? Um, our SEC Insights is a full stack, uh, LLM chatbot application-

**Swyx** [49:32]
Yeah

**Jerry Liu** [49:32]
... um, that does analysis over your SEC 10-K and 10-Q filings, uh, I think. And, and so, um, the, the goal for building this project is really twofold. Um, the reason we started building this was, one, it was a great way to dog food the production readiness for our library.

Um, we actually ended up, like, adding a bunch of stuff and fixing a ton of bugs because of this, and, and I think it was great because like-

**Swyx** [49:54]
Yeah

**Jerry Liu** [49:54]
... you know, thinking about how we handle, like, callbacks, streaming, um, actually generating, like, reliable sub responses and bubbling up sources citations, these are all things that, like, you know, if you're just building the library in isolation, you don't really think about, but if you're trying to tie this into a downstream application, like it really starts mattering, right?

**Swyx** [50:10]
Is this for your error messages? Like when you, when you talk about bubbling up stuff for observability.

**Jerry Liu** [50:13]
So, like sources, like if you-

**Swyx** [50:14]
Yeah, yeah

**Jerry Liu** [50:14]
... go into SEC Insights and you type something, you can actually see the highlights in the, in the right side. Um-

**Swyx** [50:18]
Yeah

**Jerry Liu** [50:19]
... and so, like, yeah, that, that was something that, like, took a little bit of like, um, understanding to figure out how to build well. And so it was great for dog fooding improvement of the library itself. And then as we're building the app, um, the second thing was we're starting to talk to users and just, like, trying to showcase, like, kind of, uh, bigger companies, like the potential of LlamaIndex as a framework.

Um, because these days, obviously building a chatbot, right, with Streamlit or something, it'll take you, like, 30 minutes to an hour. Like, there's plenty of templates out there on LlamaIndex, LangChain, like you can just build a chatbot. But how do you build something that kind of like satisfies some of these, uh, this, like, criteria of surfacing, like citations, being transparent, seeing like, uh, having a good UX, um, and then also being able to handle different types of questions, right?

**Swyx** [50:57]
Mm.

**Jerry Liu** [50:57]
Like more complex questions that compare different documents. That's something that I think people are still trying to explore, and so what we did was, like, we showed both like, uh, like, well, first, like organizations, the possibilities of like what you can do when you actually build something like this.

And then after, like, you know, we kind of like stealth launched this for fun just as a separate project, uh, just to see if we could get feedback from users who were using this tool to see like, you know, how we can improve stuff.

And then we were thought, we thought like, "Ah, you know, um, we built this, right? Obviously, we're not gonna sell like a financial app. Like, that's not really our, in our wheelhouse, uh, but we're just gonna open source the entire thing."

And so that now is basically just like a really nice, like, full stack app template you can use and customize on your own, right? To build your own chatbot, whether it is over, like, financial documents or over, like, other types of documents.

Um, and it provides like a nice template for basically anybody to kind of like go in and, and get started. Um, there's certain components though that, like, aren't released yet that we're going, going to in the next few, um, next few weeks.

Like one is just like kind of more detailed guides on like different modular components within it, so if you're like a full stack developer, you can go in and actually take the pieces that you want and actually kind of build your own custom flows.

The second piece is, like, take... There's like certain components in there that might not be directly related to the LLM app that would be nice to just like have people use. Uh, an example is the PDF viewer, like the PDF viewer with like citations.

I think we're just gonna give that, right? So, you know, you could be using any library you want, but then you can just, you know, just drop in a PDF viewer, right? So that it's just like a fun little module that you could, you can plug in.

**Swyx** [52:24]
Nice. Nice. Yeah. That, that's a, that's really good community service right there.

**Jerry Liu** [52:27]
Mm-hmm.

**Swyx** [52:27]
Well, so I wanna, I wanna talk a little bit about like your cloud offering, um, 'cause you mentioned, I forget the name f- that you, you had for it. Enterprise something.

**Jerry Liu** [52:35]
Um, well, one, we haven't come up with a name.

**Swyx** [52:37]
Oh, okay.

**Jerry Liu** [52:37]
Uh, we're ki- kind of calling it LlamaIndex, um, Platform, uh, Platform LlamaIndex Enterprise. Um, open to suggestions here.

**Swyx** [52:44]
Mm.

**Jerry Liu** [52:44]
Um, so I think, um, I think the high level of what I can, um, probably say is just like, like, yeah, I think we're looking at ways of like actively kind of complementing the developer experience, like building LlamaIndex.

Um, you know, we've always been very focused on stuff around like, uh, plugging in your data into the language model, and so can we build tools that help like augment that experience beyond the open source library?

**Swyx** [53:09]
Mm.

**Jerry Liu** [53:10]
Right? And so I think what we're gonna do is like, make, build an experience where it's very seamless to transition from the open source library, um, with like a one-line toggle. You can basically get this, like, complimentary service and then figure out a way to, like, monetize, uh, it a bit.

I think our, our revenue focus this year is kind of, um, is, uh, less emphasized. Like, it's more just about, like, can we build some managed offering that, like, provides complimentary value to what-

**Swyx** [53:33]
Mm

**Jerry Liu** [53:33]
... the open source library provides.

**Swyx** [53:35]
Yeah. I think it's, it's the classic thing about all open source is you wanna start building the most popular open source projects in your category to own that category. Uh, you're gonna make it e-very easy to host, therefore, then you have to...

You, you've, you've just built your biggest competitor, which is you.

**Jerry Liu** [53:52]
Yeah.

**Swyx** [53:52]
It'll be fun.

**Jerry Liu** [53:52]
I think it'll be like complimentary, 'cause I think it, it'll be like, you know, use the open source library, and then you have a toggle, and all of a sudden, you know, you can see this, um, basically like a pipeline-ish thing, um, uh, pop up, and then it'll be able to, uh, kind of like you'll have a UI.

Um, there'll be some en-enterprise guarantees, and the end goal would be to help you build like a production RAG app more easily.

**Swyx** [54:13]
Yeah.

**Jerry Liu** [54:13]
Yeah.

**Swyx** [54:13]
Great. Awesome. Um, shall we go on to-

**Jerry Liu** [54:16]
Cool

**Swyx** [54:16]
... like ecosystem and other stuff?

**Jerry Liu** [54:17]
Yeah.

**Swyx** [54:18]
Go ahead.

**Alessio** [54:18]
Um, data lo- data loaders, there's a lot of them. Uh, what are maybe some of the most popular, maybe under... not underrated, but like under expected, you know? And how has the open source side of it helped with, like, getting a lot more connectors?

### Ecosystem

**Alessio** [54:34]
You only have six people on the team today, so you couldn't have done it all yourself, I'm sure.

**Jerry Liu** [54:37]
Oh, for sure. Yeah, I think the nice thing about, like, LlamaHub itself is just it's supposed to be a community-driven hub. Um, and so actually, the bulk of the PRs are completely com- community contributed. Um, and so we, um, haven't written that many, like, first party connectors actually for this.

It's more just like, uh, kind of encouraging people to, to, uh, contribute to the community. Um, in terms of the most popular tools, uh, or the data loaders, I think we have Google Analytics on this, and I forgot the specifics.

It's some mix of, like, the PDF loaders. Um, we have like 10 of them-

**Alessio** [55:06]
Mm

**Jerry Liu** [55:06]
... but there's, uh, some subset of them that are popular. And then there's, um- ... uh, Google, like I think Gmail and like G Drive. Um, and then I think maybe it's like one of Slack or Notion. Um, one thing I will say, though, uh, and I think, like Swix might, probably knows this better than I do given that you always used to work at Airbyte, is like it's very hard to build, like especially for a full-on service like Notion, Slack, or like Salesforce, to build like a really, really high quality loader that really extracts all the information that people want, right?

**Swyx** [55:33]
Mm.

**Jerry Liu** [55:33]
Um, and so I think the thing is, um, when people start out, like they will probably use these loaders, and it's a great tool to get started. And for a lot of people, it's like good enough, and they submit PRs if they want more additional features.

If like you get to a point where you actually wanna call like an API that hasn't been supported yet, or, you know, you want to kind of, um, uh, load in stuff that like in metadata or something that hasn't been directly baked into the logic of the loader itself, uh, people start adding up, like writing their own custom loaders, and that is a thing that we're seeing.

Uh, and that's something that we're okay with, right? 'Cause like a lot of this is more just like community driven, and if you wanna submit a PR to improve the existing one, you can. Otherwise, you can create your own custom ones.

**Swyx** [56:10]
Yeah. And, and all that is, custom loader is all supported within LlamaIndex, or do you pair it with something else?

**Jerry Liu** [56:15]
Uh, oh, it's just like, I mean, um, you just define your own subclass. I think, I think that's it.

**Swyx** [56:19]
Yeah.

**Jerry Liu** [56:19]
Yeah, yeah. Uh...

**Swyx** [56:20]
'Cause, um, typically in the data ecosystem w- with Airbyte, um, you know, Airbyte has its own strategies with custom loaders, but also you could write your own with like Daxter or like Prefect or one of those tools.

**Jerry Liu** [56:31]
Yeah. Yeah, exactly. So I think for us it's more we just have a very flexible like document abstraction-

**Swyx** [56:35]
Yeah

**Jerry Liu** [56:35]
... that can fill in with any content that you want.

**Swyx** [56:37]
Okay. Um, are people really dumping all their Gmail into these things?

**Jerry Liu** [56:42]
Um-

**Swyx** [56:42]
You said Gmail is number two.

**Jerry Liu** [56:44]
Yeah. It's like one of, uh, Google, some Google product. I think it's-

**Swyx** [56:48]
Oh, it's not Gmail.

**Jerry Liu** [56:49]
I think it's Gmail.

**Swyx** [56:49]
Oh.

**Jerry Liu** [56:49]
I, I think it might be.

**Swyx** [56:50]
Oh, wow. Okay.

**Jerry Liu** [56:51]
Uh, I'm not sure, actually.

**Swyx** [56:53]
I mean, it's, that's these, you know, that's the most private, um, data source.

**Jerry Liu** [56:57]
That's true.

**Swyx** [56:59]
So I'm surprised that people dump into... I mean, I'm sure some, some people are, but like I'm sure, uh, surprised it's popular.

**Jerry Liu** [57:04]
Yeah. Let me revisit the Google Analytics.

**Swyx** [57:06]
Okay.

**Jerry Liu** [57:07]
I wanna try and give you, give you the accurate response. Yeah.

**Swyx** [57:09]
Yeah. Um, well, and, and then so, uh, the LLM engine, uh, I assume OpenAI's gonna be a majority. Is it an overwhelming majority? Uh, how, what's the market share between like OpenAI, Cohere, Anthropic, you know, whatever you're seeing, open source too?

**Jerry Liu** [57:25]
OpenAI has the majority, but then like there's Anthropic, and there's also, um, open source. I think there's a lot of people trying out like Llama 2-

**Swyx** [57:32]
Yeah, yeah

**Jerry Liu** [57:32]
... um, and, and, um, some variant of like a top open source model.

**Swyx** [57:35]
Side note, any confusion there, Llama 2 versus Llama index?

**Jerry Liu** [57:38]
Yeah, I think w- whenever I go to these talks, I always open it up with like, "We started before-"

**Swyx** [57:42]
We are not...

**Jerry Liu** [57:43]
Yeah, exactly. We started before Meta, right? I wanna point that out. Um, but no, uh, Pro- Process them, we try to use it for like branding. We just add two llamas when we have like a Llama 2 integration instead of one llama.

Uh, anyways. Uh, so, so the, um, uh, yeah, so, so I think a lot of people are trying out the popular open source models. Uh, and we have... These days we have like, um, there's a lot of toolkits and open source projects that allow you to self-host and deploy Llama 2.

**Swyx** [58:07]
Mm. Yes.

**Jerry Liu** [58:08]
Um, like, right, and, and like Ollama's just a very recent example, I think-

**Swyx** [58:11]
Yeah

**Jerry Liu** [58:11]
... that we, we added integration with. And so we just, uh, by virtue of having more of these services, I think more and more people are trying it out. Yeah.

**Swyx** [58:17]
Yeah. Do you think there's, there's potential there? Is like, um, is that gonna be an increasing trend? Like-

**Jerry Liu** [58:22]
Open source?

**Swyx** [58:23]
Yeah.

**Jerry Liu** [58:23]
Yeah, definitely. I think in general, people hate monopolies, and so-

**Swyx** [58:26]
Yeah

**Jerry Liu** [58:26]
... um, like there's a... Whenever like OpenAI has something really cool or like any, um, company has something really cool, even Meta, like there's g- just gonna be a huge competitive pressure from other people to do something that's more open and better.

Um, and so I do think just market pressures will, will im- improve, like-

**Swyx** [58:41]
Yeah

**Jerry Liu** [58:42]
... open sourced option.

**Swyx** [58:43]
Last thing I'll say about this, which, which is just really like, it's, it gets clicks. It's people like are like psychologically want that, but then at the end of the day, they want, they fall for brand name and popular...

and, and performance benchmarks, you know? Um, and at, at the end of the day, OpenAI still wins on, on that.

**Jerry Liu** [59:00]
I think that's true. Um, but I, I just think like, um, unless you're like an active employee at OpenAI-

**Swyx** [59:04]
Yeah

**Jerry Liu** [59:04]
... right, like all these research labs are putting out like ML, like PhDs or, or kind of like other companies too-

**Swyx** [59:10]
Yeah

**Jerry Liu** [59:11]
... that are investing a lot of dollars. Uh, there's just gonna be a lot of like competitive pressures to develop like better models. So is it gonna be like all fully open source with like a permissive license? Like, I'm not completely sure, but like there's just a lot of just incentive for people to develop better stuff here.

**Swyx** [59:23]
Yeah. Have you looked at like RAG specific models, like Contextual?

**Jerry Liu** [59:27]
Um, no. Is it public or?

**Swyx** [59:29]
No. They, they literally just, uh, so Duy Guilla, I think is, is his name, and you probably came across him. Um, he wrote the RAG paper at Meta.

**Jerry Liu** [59:38]
Mm-hmm. Mm-hmm.

**Swyx** [59:38]
Um, and just started Contextual AI to create a RAG specific model. I don't know what that means. I was hoping that you do 'cause it's your business.

**Jerry Liu** [59:47]
Yeah. I mean, um, if I had insider information. I mean, you know, to be honest, I think this, this kind of relates to my previous point on like RAG and fine-tuning.

**Swyx** [59:56]
Mm.

**Jerry Liu** [59:56]
Like a RAG specific model is a model architecture that's designed for better RAG.

**Swyx** [1:00:00]
Yes.

**Jerry Liu** [1:00:00]
Um, and it's less the software engineering principle of like, how can I take existing stuff and just plug and play different components into it?

**Swyx** [1:00:06]
That's it. Yeah.

**Jerry Liu** [1:00:06]
Um, and there's a beauty in that from ease of use and modularity, but, um, like when you wanna end-to-end optimize the thing, you, you might want a more specific model. Um, I just, yeah, I don't know. I, I think, I think building your own models is honestly pretty hard.

Um, and I think the issue is if you also build your own models, like you're also just gonna have to keep up with like the rate of LLM advances. Like how, like basically the question is, when GPT-5 and 6 and whatever, like Anthropic, Claude 3 comes out, like what's...

How can you prove that you're actually better than a software developer just cobbling together their own components-

**Swyx** [1:00:38]
Mm

**Jerry Liu** [1:00:38]
... on top of a base model, right?

**Swyx** [1:00:39]
Yeah.

**Jerry Liu** [1:00:39]
Even if just like conceptually, this is better than maybe like GPT-3 or GPT-4.

**Swyx** [1:00:44]
Yeah. Yeah. Base model game is expensive.

**Jerry Liu** [1:00:47]
Yeah.

**Alessio** [1:00:48]
Um, what about vector stores? I know, uh, Swix is wearing a, a Chroma sweatshirt.

**Swyx** [1:00:52]
Yeah, they got the, they in the swag game.

**Jerry Liu** [1:00:54]
I have the, I have the mug, uh, from Chroma.

**Swyx** [1:00:56]
Yeah.

**Alessio** [1:00:56]
Yeah.

**Jerry Liu** [1:00:56]
It's great. Yeah.

**Alessio** [1:00:57]
Yeah. Um, what do you think, what do you think there? Like there's a lot of them. Are they pretty interchangeable for like your users' use case? Uh, is HNSW all we need? Is there room for improvements there?

**Swyx** [1:01:09]
Is MPRA all we need?

**Alessio** [1:01:11]
Yeah, yeah. It's the-

**Jerry Liu** [1:01:12]
I think, yeah, we try to remain unopinionated about storage providers, so it's not like we, we don't try to, like, play favorites, so we have, like, a bunch of integrations obviously, and we-- The way we try to do it is we just try to find, like, some standard interfaces, but obviously, like, different vector stores will support kind of like, uh, slightly additional things like metadata filters and those things.

Um, and the goal is to have our users basically, uh, leave it up to them to try to figure out, like, what makes sense for their use case. Um, in terms of, like, the algorithm itself, um, I don't think the delta on, like, improving the vector store, like, embedding lookup algorithm is that high.

I think the stuff has been mostly solved, um, or at least there's just a lot of other stuff you can do, um, to try to improve the overall performance.

**Swyx** [1:01:48]
Like what? Like what?

**Jerry Liu** [1:01:49]
No, I mean, like, everything else that we just talked about. Like-

**Swyx** [1:01:51]
Okay

**Jerry Liu** [1:01:51]
... uh, in terms of, like, accuracy, right? To improve RAG, like, everything that we talked about, like-

**Swyx** [1:01:54]
How's it-

**Jerry Liu** [1:01:54]
... chunking, like, metadata, like, um-

**Swyx** [1:01:56]
Yeah. Well, I mean, I was just thinking, like-

**Jerry Liu** [1:01:58]
Yeah

**Swyx** [1:01:58]
... uh, may- maybe the, for me, the inqu- interesting question is, you know, there are, like, eight... It's like kind of Game of Thrones. There's, like, eight-- the war of eight databases right now.

**Jerry Liu** [1:02:06]
Oh, oh, I see what you're saying.

**Swyx** [1:02:06]
Um, how do they stand out, and how do they become very good partners with LlamaIndex?

**Jerry Liu** [1:02:10]
Um. Oh, I mean, I think we're, yeah, we're pretty good partners with, with most of them. Uh, let's see. Um-

**Swyx** [1:02:16]
Well, like, so if you're a, you know, a vector database founder, like, what do you, what do you work on?

**Jerry Liu** [1:02:21]
That's a good question. I think one thing I'm very interested in is, um... And this is something I think I've, I've started to see a general trend towards, is combining structured data querying with unstructured data querying.

**Swyx** [1:02:33]
Mm.

**Jerry Liu** [1:02:34]
Um, and I think that will probably just, um, expand the query sophistication of these vector stores, um, and basically make it so that users don't have to think about whether they separate those out.

**Swyx** [1:02:42]
Would you call this, like, hybrid querying? Is that what Weaviate's doing?

**Jerry Liu** [1:02:47]
Yeah.

**Swyx** [1:02:47]
Is doing?

**Jerry Liu** [1:02:47]
I mean, I think, like, if you think about metadata filters, that's basically a structured filter. It's like a select star or select where, uh-

**Swyx** [1:02:53]
Yeah

**Jerry Liu** [1:02:53]
... right, um, something equals something, and then you combine that with semantic search.

**Swyx** [1:02:57]
Yeah.

**Jerry Liu** [1:02:57]
I know, I think, like, LanceDB or, or something was, like, uh, tried, was trying to do some, like, joint interface. Um, the reason is, like, most data is semi-structured. There are some structuring annotations, and there's some, like, unstructured text.

And so, like, um, somehow combining all the expressivity of, like, SQL with, like, the flexibility of semantic search is something that I think is gonna be really important, right? And we, we have some basic hacks right now that allow you to q- jointly query both a SQL database and, like, a separate SQL database and a vector store to, like, combine the information, but that's obviously gonna be less efficient than if you just combined it into one system.

Yeah. And so I think, like, pgvector, like, you know, that type of stuff, like, I think it, it's starting to get there. But, like, in general, like, how do you have an expressive query language to actually-

**Swyx** [1:03:36]
Mm

**Jerry Liu** [1:03:37]
... do, like, structured querying along with, like, all the capabilities of semantic search?

**Swyx** [1:03:40]
So your current favorite is just put it into Postgres?

**Jerry Liu** [1:03:44]
Uh, no, no, no. I, we, we don't play-

**Swyx** [1:03:46]
Is it the, the Post- Postgres language? The, the-

**Jerry Liu** [1:03:48]
Um-

**Swyx** [1:03:48]
... the query language.

**Jerry Liu** [1:03:50]
I actually don't know what the best language would be for this-

**Swyx** [1:03:54]
Right

**Jerry Liu** [1:03:54]
... 'cause I think the, it'll be something that, like, the model hasn't been fine-tuned over. Um, and so you, you might want to train the model over this.

**Swyx** [1:04:00]
Yeah.

**Jerry Liu** [1:04:00]
But some way of, like, expressing, um, structured data filters.

**Swyx** [1:04:05]
Yeah.

**Jerry Liu** [1:04:05]
Um, and this could be, include time too, right? It could, it doesn't have to just be, like, a where clause, uh, with this idea of, like, uh, semantic search. Yeah.

**Swyx** [1:04:12]
Yeah, yeah. And we talked about, uh, graph representations.

**Jerry Liu** [1:04:15]
Yeah. Oh, yeah. That's another thing too.

**Swyx** [1:04:16]
Which is something that you-

**Jerry Liu** [1:04:16]
And there's, like... And, and yeah, so that's actually something I, I didn't even, um, bring up yet. Like, there's this interesting idea of, like, can you actually have the language model, like, explore, like, relationships within the data too, right?

And somehow combine that information with stuff that's, like, more, more, um, structured within the DB.

**Swyx** [1:04:31]
Awesome.

**Alessio** [1:04:33]
Um, what else is left in the stack?

**Swyx** [1:04:35]
Oh, uh, evals.

**Alessio** [1:04:36]
Yeah.

### Evals & Agents

**Swyx** [1:04:36]
Um, what are your current strong belief about how to evaluate RAG?

**Jerry Liu** [1:04:41]
I think I have thoughts. I, I think we're trying to curate this into some, like, more opinionated principles, um, because there are some, like, open questions here. I think one question I had to think about is whether you should do, like, uh, evals, like, component by component first, or you just do the end-to-end thing.

Um, I think you should... You might actually just wanna do the end-to-end thing first, just to do a sanity check of whether or not, like, this-

**Swyx** [1:05:00]
Mm

**Jerry Liu** [1:05:00]
... uh, given a query and the final response, whether or not it even makes sense. Like, you eyeball it, right? And then you, like, try to do some basic evals. Um, and then once you, like, diagnose what the issue is, then you go into the kind of like, specific area to, to find some more, uh, solid benchmarks and try to, like, improve stuff.

Um, so what is end-to-end evals? Like, it's, you, um, have a query. It goes in through retri- uh, uh, retrieval system. You get back something. You synthesize a response, and that's your final thing, and you evaluate the quality of the, the final response.

Um, and these days, there's plenty of, uh, projects, like startups, like companies, uh, research, uh, doing stuff around, like, GPT-4, right? As, like, a human judge to basically-

**Swyx** [1:05:37]
Yeah

**Jerry Liu** [1:05:37]
... kind of like synthetically generate a dataset.

**Swyx** [1:05:38]
Do you think those will do well?

**Jerry Liu** [1:05:39]
I mean, I think-

**Swyx** [1:05:40]
It's too easy.

**Jerry Liu** [1:05:42]
Well, I think... Oh, oh, you're talking about, like, the startups? Um-

**Swyx** [1:05:45]
Yeah

**Jerry Liu** [1:05:45]
... I don't know. I don't know from the startup side. I just know from the technical side, I think, I think people are gonna do more of it. Um, the main issue right now is just, uh, it's really unreliable.

Like, it's, it's just, uh-

**Swyx** [1:05:55]
Oh, well

**Jerry Liu** [1:05:55]
... like, the, there's, like, variance on the response whenever, whenever you run it through-

**Swyx** [1:05:57]
Yeah. Then they won't do more of it. I mean, 'cause it's bad. It's a bad eval.

**Jerry Liu** [1:06:00]
No, but these models will get better, and you'll probably fine-tune the models to be a better judge. I think that's probably what's gonna happen. So I'm, like, reasonably bullish on this because I don't think there's really a good alternative beyond you just human annotating a bunch of datasets- ...

um, and then trying to, like, just manually go through and, and curating and, like, validating eval metrics.

**Swyx** [1:06:17]
Okay. And-

**Jerry Liu** [1:06:18]
Uh, so this is just gonna be a more scalable solution.

**Swyx** [1:06:20]
Mm-hmm.

**Jerry Liu** [1:06:20]
In terms of the startups, yeah, I mean, I think there's a bunch of companies doing this. In the end, it probably comes down to some aspect of, like, UX speed, uh, and then, uh, whether you can, like, fine-tune a model.

Um-

**Swyx** [1:06:29]
Yeah

**Jerry Liu** [1:06:29]
... and then, uh, so that's end-to-end evals. And then I think, like, what we found is for RAG, a lot of times, like, uh, what ends up affecting this, like, end response is retrieval. You're just not able to retrieve the right response.

I think having proper retrieval benchmarks, especially if you wanna do production RAG, is, is actually quite important. I think what does having good retrieval metrics tell you? It tells you that at least, like, the retrieval is good. It doesn't necessarily guarantee the end generation is good, but at least it gives you some sort of, like, um, uh, sanity check, right?

So you can, like, fix one component while optimizing the rest. What retrieval, like, evaluation is pretty standard, and it's been around for a while. It's just like an IR problem, basically. You have some, like, uh, input query. You get back some retrieves out of context, and then there's some ground truth in that ranked set, and then you try to measure it based on ranking metrics.

So the closer that ground truth is to the top, the more you reward the evals. And then the closer it is to the bottom, or if it's not in, in the retrieved set at all, then you penalize the evals.

Um, and so that's just, like, a classic ranking problem. Most people who are starting out probably don't know how to do this.

**Swyx** [1:07:32]
Mm.

**Jerry Liu** [1:07:33]
Uh, right now we, we just launched some, like, basic retrieval evaluation, um, modules that help users do this. Uh, one is just, like, curating this data set in the first place, and one thing that we're very interested in is this idea of, like, synthetic data set generation for evals.

So how can you, given some context, generate a set of questions with GPT-4, and then all of a sudden you have, like, question and then context pairs, and that becomes your ground truth.

**Swyx** [1:07:54]
Yeah. Are data agent evals the same thing, or, uh, uh, is there a separate set of stuff for agents that you think is relevant here?

**Jerry Liu** [1:08:02]
Data agents add, like, another layer of complexity 'cause then it's just like you have just more loops in the system. Like, you, um, can evaluate, like, each chain of thought loop itself, like every LLM call, to see whether or not the input to that specific step in the chain of thought process actually, um, uh, works, uh, or, or is, is correct.

Uh, or you could evaluate, like, the final response to see if that's correct. This gets even more complicated when you do, like, multi-agent stuff because now you have, like, some communication between, like-

**Swyx** [1:08:28]
Yeah

**Jerry Liu** [1:08:28]
... different agents. Like, you have a top-level orchestration agent passing it on to some, uh, low-level stuff. Um, I'm probably less, uh, familiar with kind of, like, agent eval frameworks. I, I know they're, they're starting to be- become a thing.

Um, I know I was ta- I was, uh, talking to, like, June from the Journal of Agents paper, uh, which is pretty, pretty unrelated to what we're doing now, but it's very interesting, where it's like, so you can kind of evaluate, like, overall agent simulations by just, like, kind of understanding whether or not they, like, model the distribution of human behavior.

But that's at-

**Swyx** [1:08:56]
Oh

**Jerry Liu** [1:08:56]
... like, a very macro principle, right? And that's very much to evaluate stuff, to kind of, like, model the distribution of things. Um, and, and I think that works well when you're trying to, like, generate something for, like, creative purposes.

But for stuff where you really want the agent to, like, achieve a certain task, it really is, like, whether or not it achieved the task or not, right? 'Cause then it's not like, "Oh, does it generally mimic human behavior?"

It's like, "No, like, did you, like, send this email or not?" Right? Like, because otherwise, like, this, this thing didn't work. Yeah.

### Lightning Round

**Swyx** [1:09:22]
Makes sense.

**Alessio** [1:09:23]
Awesome. Um, yeah, let's jump into lightning round. Uh, so we have three question: acceleration, exploration, and then one final takeaway. The acceleration question is, what's something that already happened in AI that you thought would take much longer to get here?

**Jerry Liu** [1:09:38]
I think just the ability of LLMs to generate believable outputs, um, and, and both, uh, for text and also for images. And I think just, um, the, the whole reason I started hacking around with LLMs, honestly, I felt like I got into it pretty late.

I actually got into it, like, early 2022, 'cause GPT-3 had been out for a while. Like, just the fact that, um, there was this engine that was capable, like, reasoning, and no one was really, like, tapping into it.

Um, and then the fact that, uh, you know, I used to work in image generation for a while. Like, I, I did GANs and stuff back in the day, and that was, like, pretty hard to train. You would generate these, like, 32-by-32, uh, images, and then now taking a look at some of the stuff by, like, DALL-E and, and, you know, Midjourney and those things, so it's, it's just, it's, it's very good.

Yeah.

**Swyx** [1:10:17]
Exploration. Um, what do you think is the most interesting unsolved question in AI?

**Jerry Liu** [1:10:22]
Yeah. I'd probably work on some aspect of, um, like, personalization of memory. I, I think a lot of people have thoughts about that, but, like, for what it's worth, I don't think the final state will be RAG. I, I think it will be some, some, like, fancy algorithm or architecture where you, like, bake it into, like, the, the architecture of the model itself.

Like, if, if you have, like, a personalized assistant that you can, um, talk to that will, like, learn behaviors over time, right, and, and kind of, like, learn stuff through, like, conversation history, what, what exactly is the right architecture there?

I, I do think that'll be part of, like, the neural, neural net.

**Swyx** [1:10:53]
Like continuous fine-tuning?

**Jerry Liu** [1:10:54]
Yeah, like some aspect of that. Right, right, like these are... Like, I don't actually know the specific technique-

**Swyx** [1:10:58]
Mm

**Jerry Liu** [1:10:58]
... but I don't think it's just gonna be something where you have, like, a fixed vector store and that, that thing will be, like, the thing that restores all your memories.

**Swyx** [1:11:04]
Yeah. It, it's interesting because I feel like using, um, model weights for memory is just such an unreliable storage device.

**Jerry Liu** [1:11:14]
I know. But, like, I, I just think, uh, from, like, the AGI, like, you know, just modeling, like, the human brain perspective, I think that, uh, there is something nice about just, like, being able to optimize that system, right?

And to optimize a system, you need parameters, and then that's where you just get into the neural net piece.

**Swyx** [1:11:31]
Cool.

**Alessio** [1:11:32]
Cool. Uh, and yeah, takeaway. You got the audience ear. What's something you want everyone to think about or, yeah, take away from this conversation and your thinking?

**Jerry Liu** [1:11:42]
I think there were a few key things. Uh, so we talked about two of them already, which was SEC Insights, which if you guys haven't checked it out, I would definitely encourage you to do so, because it's not just, like, a random, like, SEC app.

It's, like, a full stack thing that we open sourced, right? And so if you guys wanna check it out, I would, uh, definitely do that. It provides a template for you to build kind of, like, production grade RAG apps.

Um, and we're gonna open source, like, and modularize more components of that soon.

**Swyx** [1:12:04]
And do a workshop on it.

**Jerry Liu** [1:12:05]
Um, yeah. And, and the second piece is we are thinking a lot about, like, retrieval and evals. Um, I think right now we're kind of exploring integrations with, like, a few different partners, and so s- hopefully some of that will be, uh, released soon.

And so just, like, how, how do you basically, um, have an experience where you just, like, write LlamaIndex code. All of a sudden you can easily run, like, retrievals, evals, and, like, traces, all that stuff in, in, like, a service.

And so I think we're working with, like, a few providers on that. Um, and then the other piece, which we did talk about already, is this idea of, like, yeah, building, like, RAG from scratch. I mean, I think everybody should do it, I think.

Um, uh, like, I would check out the guide if you guys haven't already. I think it's in our docs. But, um, instead of just using, um, you know, either the kind of like, uh, the retriever query engine in LlamaIndex or, like, the conversational, like, QA train in, in LangChain, it's, um, I would take a look at how do you actually chunk parse data and do, like, top-k embedding retrieval, 'cause I really think by doing that process, it helps you understand the decisions, the prompts, the language models to use.

Um.

**Swyx** [1:13:04]
That's it.

**Jerry Liu** [1:13:05]
Yeah.

**Swyx** [1:13:05]
Thank you so much.

**Alessio** [1:13:06]
Thank you, Jerry.

**Jerry Liu** [1:13:06]
Yeah, thank you.

---

This library is powered by PodHood (https://podhood.com), the podcast website platform.