# [Paper Club] Writing in the Margins: Chunked Prefill KV Caching for Long Context Retrieval Latent Space · 2024-09-19 Umar Jamil from Writer presents 'Writing in the Margins,' a paper on improving long-context retrieval in LLMs by leveraging chunked prefill of the KV cache. The method splits a long prompt into chunks, generates query-relevant annotations (margins) after each chunk, and appends them at the end before generating the final answer. This approach solves the 'lost in the middle' problem without fine-tuning, and unlike traditional RAG or separate summarization, it avoids re-prefilling the entire context, reducing cost from double to single prefilling. The technique is compatible with any transformer model and includes overlapping margin generation and classification within the same batch request. Benchmarks show consistent improvements across models, and the implementation is open-source on GitHub. ## Questions this episode answers ### What is the 'Writing in the Margins' technique for long-context retrieval? Umar Jamil explains that Writing in the Margins exploits chunked prefill—a method of incrementally loading long prompts into the KV cache—to generate extractive summaries (margins) relevant to the user's query after each chunk. These margins are then appended before the final question, making key information prominent at the end of the prompt to improve retrieval, without any model fine-tuning. [8:54](https://addtry.com/508b86f5-1b3e-4749-a59b-f7cca90d4315?t=534000) ### How does Writing in the Margins reduce inference cost compared to naive chunking? Umar Jamil notes that with traditional chunking, you pay twice the prefilling cost: once when processing each chunk separately and again when feeding the full context plus summaries to the model. Writing in the Margins avoids the second prefilling by generating margins on-the-fly; processing a 1‑million‑token document costs roughly 1 million tokens versus 2 million with the naive approach. [16:01](https://addtry.com/508b86f5-1b3e-4749-a59b-f7cca90d4315?t=961000) ### Is Writing in the Margins model‑specific, or does it work with any LLM? Umar Jamil states it is compatible with any off‑the‑shelf transformer model without any fine‑tuning. It only modifies the inference pattern—how the KV cache is populated during chunked prefilling—not the model architecture or weights. The paper shows improvements across various pre‑trained models, particularly benefiting smaller ones. [22:53](https://addtry.com/508b86f5-1b3e-4749-a59b-f7cca90d4315?t=1373000) ### How does Writing in the Margins address the 'lost in the middle' problem? According to Umar Jamil, the 'lost in the middle' phenomenon means LLMs retrieve information better from the start or end of a prompt. Writing in the Margins generates query‑relevant summaries during chunked prefilling and places them immediately before the final question, effectively moving key insights to the end, which boosts retrieval accuracy without fine‑tuning. [12:19](https://addtry.com/508b86f5-1b3e-4749-a59b-f7cca90d4315?t=739000) ## Key moments - **[0:00] Intro** - **[1:35] KV Cache** - [1:35] Writer presents 'Writing in the Margins' paper on better inference pattern for long context retrieval - [3:07] Prefilling prompts in KV cache has quadratic cost in memory and compute, making it the most expensive LLM operation - **[4:50] Long Context** - [5:28] Chunked Prefill splits long prompts into fixed-size chunks because full prefill is computationally prohibitive - [7:57] Writing in the Margins generates annotations from partial KV cache, analogous to a student taking notes while reading a book - **[8:15] Writing Margins** - [8:54] Writing in the Margins process: prefill chunk, append extractive prompt, generate margin, delete prompt and margin, then next chunk - [9:29] KV cache deletion is only possible from the end due to autoregressive causal dependency - [10:57] Margin classification can be overlapped with margin generation in the same inference request, avoiding another batch - [11:33] Margins are appended at the end to exploit LLMs' position bias toward beginning/end, solving 'lost in the middle' - **[13:16] Benchmarks** - [13:16] Writing in the Margins requires no fine-tuning and works with any off-the-shelf transformer model - [14:27] Writing in the Margins boosts smaller models more significantly than larger models - [15:04] Writing in the Margins outperforms both RAG and vanilla long-context processing on benchmarks - **[15:43] Cost** - [16:14] A 1M-token prompt costs 2M tokens with standard chunking but only 1M tokens with Writing in the Margins - **[17:45] Demo & Classification** - [17:45] LinkedIn video demo shows Writing in the Margins annotating a document with extractive summaries - [20:45] Q: Does each margin only summarize the current chunk? A: No, each margin summarizes all preceding chunks. - **[24:07] Code** - [24:21] Writer open-sourced the implementation supporting Llama, Phi, and Qwen models on GitHub - **[25:41] Q&A** - [26:20] Q: Does Writing in the Margins require fine-tuning the model? A: No, but it needs inference engine modifications for KV cache manipulation. - [32:30] Q: Can attention be restricted to only the latest chunk? A: KV cache tokens are contextualized; looking at only the latest would be out-of-distribution. - [34:54] Q: How are queries chosen for Writing in the Margins? A: The query is part of the standard prompt structure, already known. - [37:55] In benchmarks, RAG was assisted by a 70B model to judge relevance, giving it an advantage over vector similarity - **[39:11] Efficiency** - [40:07] A 1M token prompt would require a 1M x 1M attention matrix, making full prefill impossible - [42:10] Token generation scales linearly with context length, but prefill scales quadratically - [43:59] Writing in the Margins has a token generation cost, but benchmarks show it is always beneficial over plain long context - [44:55] Writing in the Margins is not beneficial when prompts fit in memory without chunking - [46:17] Ablation shows providing all context plus margins always outperforms using only extracted margins - **[48:12] Future** - [48:30] Writing in the Margins became possible only after long-context models made chunked prefill necessary - [50:17] Future research will focus on attention mechanisms like Sigmoid Attention to improve long-context utilization - [51:28] Long-context modeling is essential for analyzing full books or codebases, moving beyond retrieval-based RAG - **[51:45] Outro** ## Speakers - **Eugene** (host) - **Vibhu** (host) - **Umar Jamil** (guest) ## Topics Inference, Language Models ## Mentioned Writer (company), LangChain (product), Llama (product), Phi (product), Qwen (product), vLLM (product) ## Transcript ### Intro **Vibhu** [0:00] It's recorded, but TLDR probably just shared internal. So Sam, we'll, we'll get you a recording after. Cool. I'm, I'm gonna pass off to you guys whenever you wanna start. I'm sure people will trickle in. **Umar Jamil** [0:19] So from Writer side, I will be presenting. I don't know if you guys can hear me. **Vibhu** [0:24] Yep. **Umar Jamil** [0:26] All right. Uh, there should be also Wasim. I don't know if he's also present, but, uh, we'll kick off the presentation. Uh, let me know whenever you, everyone is ready, and I will start. **Eugene** [0:40] I think we're good. Take it away. **Umar Jamil** [0:47] So, okay, I will start by sharing my screen. Meanwhile, uh, let's see if it works. Uh, desktop two, and let me know if you can sh- see my screen. **Eugene** [1:18] Yep, we can see your slides. **Umar Jamil** [1:22] All right, perfect. So tonight, uh, can I start right, everyone? **Vibhu** [1:29] Yeah. **Umar Jamil** [1:30] Okay. **Vibhu** [1:30] You should be good. It's recording. People might trickle in every now and then, but- **Umar Jamil** [1:33] Perfect. Perfect **Vibhu** [1:34] ... 30 in. ### KV Cache **Umar Jamil** [1:35] All right. So tonight I will be presenting a paper, uh, that came out from Writer, Writing in the Margins, uh, Writing in the Margins: Better Inference Pattern for Long Context Retrieval. Um, it's a paper that involves, uh, long context and how we leverage, uh, the KV cache to make, uh, long context modeling, uh, more, uh, effective. So I will, uh, I will do a very, m- a little technical, but not so technical presentation, and later we can deep dive into the details of the paper. So let me open the chat so I can see what everyone is writing meanwhile I'm, while I'm talking. All right. **Eugene** [2:15] Yeah, I think, I think, Umar, you can just ignore the chat. People are gonna be... **Umar Jamil** [2:19] All right. **Eugene** [2:19] There's, it's gonna be really buzzy, and there's gonna be a lot of people like, "Oh, wow, that's so cool," and everything. Yeah. **Umar Jamil** [2:24] Oh. **Eugene** [2:24] Vibhu and I will take care of the chat for you. **Umar Jamil** [2:26] Perfect. **Vibhu** [2:27] If something super crazy pops up, we'll let you know, otherwise we'll, we'll take care of it. **Umar Jamil** [2:31] Perfect. Perfect. All right. So I will skip the part of what is a language model, but okay, the language model is a probabilistic model, and, uh, that leverages the prompt to generate what is the next token and how we generate text to the language model, we do it iteratively, so one token at a time. Um, most of the language models nowadays are based on the transformer model, and in the transformer model, basically whenever we have a prompt, the first thing that we do is we put this prompt into the memory of the language model that is known as the KV cache, um, which is basically the keys and values in these transformer layers. And the operation of creating this initial KV cache is known as prefilling. Uh, so the first thing that y- the, the, the inference engine, which could be vLLM, which could be TensorRT or any other inference, uh, um, uh, framework that you're using, the first thing that it does with your prompt is doing this prefilling. And if you're interested, actually, the, the prefilling is one of the most expensive part of the language, uh, of processing a prompt, because it has a quadratic cost with respect to the computations as well to, in the, uh, with respect to the memory. So imagine that we have a prompt called, "Austin is a city in", uh, a prompt that says, "Austin is a city in..." Then the first thing that we do to generate text with this prompt is we prefill it into the language model in the KV cache, and then we leverage it to generate tokens, one token at a time. And the KV cache is, uh, this kind of memory in the language model, uh, in any actually transformer model that is autoregressive. Uh, that is, uh, uh, that if it contains some tokens, then the language model will leverage them. If it doesn't contain those tokens, then the language model cannot leverage them. So, uh, the language model only sees what is inside of the KV cache. So what happens is to our prompt, "Austin is a city in," the first thing that we do is we do this prefilling, which puts these tokens into the, um, KV cache, uh, which is one for each layer of the transformer, and then we generate what is the next token, uh, by asking the l- uh, the language model what is the next token, and suppose the next token is the word the, "Texas". We take this token, "Texas", we keep it in the KV cache so that the language model can leverage to generate the next token, and then suppose the next token is a comma, et cetera, et cetera, until we generate the f- the, the entire response of the language model. Now, we have seen that in the past few years, uh, prompts are becoming longer and longer. So we started with, uh, 2,000, uh, context window to 4,000, 8,000, 32, 64, 128. Now we have reached millions of tokens. Um, this means that you can send to, an entire book to the language model, and you can ask, uh, the language model questions about this book. ### Long Context **Umar Jamil** [5:12] However, with, uh, great prompts comes great responsibility . And, um, the, the, the reason is, uh, the following. So imagine you have a very long prompt. Suppose that you have a book and you want the language model to answer questions about this book. Uh, suppose that this book is, like, 1 million tokens long. If you r- do the prefilling of this 1 million tokens, uh, prompt in, uh, uh, in the KV cache, the language model will actually not be able to do that because, uh, as I s- as I said before, the prefilling is one of the most expensive operations that we do, uh, when inferencing language models. Uh, why? Because it's quadratic with respect to the sequence length in terms of memory and also in terms of compute. So, uh, language models actually cannot, uh, prefill the entire prompt in one single pass i- in the la- in the KV cache. So what they do is they do it by chunks. This is called Chunked Prefill, and it's, uh, an experimental feature that has been recently introduced in vLLM, but it's probably, uh, used in, uh, more sophisticated inference engines at, um, the- But, uh, uh, major companies. Uh, so what we do by, mm, with the, mm, chunked prefilling, basically we split the prompt of the user into multiple chunks and we prefill each chunk step by step. So suppose that the user sent an entire book, which is made up of 10 chapters. Now, uh, the chunk, usually they are of a fixed size. This doesn't mean that the, the chunk has some contextual meaning. It may not be the first chapter of the book or the second chapter of the book. It could be, just be the first 4,000 tokens of the prompt, and then the next 4,000 tokens of the prompt, et cetera, et cetera. So we've prefilled the first chunk into the KV cache. Then we prefill the second chunk in the KV cache. So now the KV cache contains the first chunk and the second chunk, then the third chunk, et cetera, et cetera, until all the prompt is inside of the KV cache, which can then be leveraged to generate tokens. So the... This is intuitively very similar to how a student would read a book. For example, imagine a student is given a book to read and then to, uh, answer questions, uh, uh, a question about this book. What the student would do, uh, uh, would, um, the student would read the first chapter, and now the brain of the student contains information about only the first chapter. Then the student would read the second chapter, and now the brain of the student contains information about the first chapter and the second chapter of the book, and then et cetera, et cetera, until the student reads the last chapter. Now the brain of the student contains information about all the chapters. And, uh, then we, uh, the student will be given the question, and the question, um, and then the student has to leverage the information that he has, uh, read from the book to answer this question. However, the student would struggle to do it. Why? Because intuitively, when you read a book, the, the moment you start reading the second chapter, you can have already forgetting what is, uh, the first chapter about. Um, so what is a better strategy that this could, you, this student could do? ### Writing Margins **Umar Jamil** [8:15] Well, the student could, uh, read the chapters, and while reading, it could take some annotations. And this is what we do with Writing in the Margins. So because of very long prompts, we are already forced to split the prompt into multiple chunks to do this, uh, Chunked Prefill. Why not leverage the partially prefilled KV cache to generate some annotations that can be then, then be leveraged to improve the model's capability of extracting information from this prompt? So basically, Writing in the Margins, from a technical point of view, works as follows. We have this very large prompt. We split it into chunks because we are forced to split it into chunks. We cannot prefill the entire, uh, uh, context into the KV cache. We prefill the first chunk, and then we add, after the first chunk, we add a prompt that tells the model, "Okay, use the text above to extract information about the query." And the query is, uh, the question that we want to, uh, uh, to get an answer to. So now the KV cache contains the first chunk and this prompt here, and we leverage it to generate a few tokens, which is the, this margin annotation. And then this is another trick. You can delete stuff from the KV cache, only from the end. Uh, why you can do that? Because, um, a language model is an autoregressive language mo- uh, is an autoregressive model, which means that every token depends on all past tokens, which means you can delete from, stuff from the end and regenerate it eventually. But you cannot, of course, delete stuff from the beginning or from the middle because it would invalidate all the future tokens. But you can always remove stuff from the end. So what we do is we prefill the first part of prompt. We prefill this instructive ins- instruction. We generate a few tokens, which is the margin annotation. Then we can delete this margin and this, um, instruction that we have added, and then we prefill the second chunk. Uh, we then append another prompt, um, extractive prompt. We generate a few more tokens, which is the second margin. So the second margin will depend on the first and the second chunk. Then we gen- delete these second margins tokens. We delete the extractive prompt, et cetera, et cetera, until we have processed all the prompt. Um, and this is a, um, a visualization that we have also put in the paper. So basically, uh, imagine you are given a very large, large prompt. So what we do is we prefill the first part in s- in the, in the KV cache, and then we extract information, uh, on what is present in the KV cache, and we call it the first margin. Uh, we then also can classify these margins. And, uh, in the paper, we also show that actually the, the computation and the prefilling of the margins, uh, sorry, the computation of the margin and the classification of the margin can be overlapped inside of the same request in the batch, so you don't need to have another request in the batch. But this is okay, a little KV, KV cache tricks for optimizing the inference. So what do we do with these margins? Basically, we append them at the end before asking the question to the language model because our goal is, okay, we have this very big context, and then we have some question. So what we do, instead of just asking the question, the model may not be able to find it, so we generate these margins, and then we append them right before asking the question, and then we ask the model the question. So now the model can also leverage these margins, which are present right before the question, to better answer the question. Uh, why these margins are... Do, do, do, um, why this margin will be leveraged by the language model? First of all, because they are, uh, uh, they are, um, uh, uh, the, the instruction that we use to extract them is an extractive summary. So the, we ask the language model to extract information, uh, using the prefilled KV cache on, um, knowing what is the query. So, uh, these margins are relevant to answer that particular query. Um, why do we add these margins at the end before asking the question? Because a few months ago, there was a paper called Lost in the Middle. Basically, it says that, uh, and it's actually true. Uh, basically, it says that, um, if the relevant information that you are trying to extract from this prompt, m- m- which could be very large, is present either at the beginning or at the end of the prompt, then the language model will very likely be able to... will be- very likely will be able to find it. However, if this, if this information is present kind of in the middle, then the l- l- the language model will less likely, uh, be able, uh, to find it. So that's why we add them to the end, so it improves the, uh, language model's ability to, to, to leverage this information. But does it even work? So yes, we have proof, and, uh, we have... we show in the paper, uh, a comparison of, uh, pre-trained language models. So first of all, we are not fine-tuning any language model. We are not changing anything. This is just a different way of utilizing something that is already being done, which is the Chunked Prefill of the KV cache, to improve a language model's ability to leverage long context. ### Benchmarks **Umar Jamil** [13:32] So it can be used with any transformer model without fine-tuning, just by doing it diff, uh, just by doing this inference differently. That is, don't just blindly prefill the KV cache, but prefill chunk by chunk because you are forced to, and then leverage it to extract these margins, and then leverage all these extracted margins at the end to- **Guest** [13:51] Uh, to clarify and make sense of, I'm just writing code for you and building it. What you're asking me to do... **Umar Jamil** [13:57] Sorry? **Vibhu** [14:00] I think someone had mic unmuted. **Eugene** [14:02] Accident- **Vibhu** [14:02] I just muted them **Eugene** [14:03] ... accidental. Yeah. **Vibhu** [14:05] Go ahead. **Umar Jamil** [14:06] Okay. Um, so yeah, in the paper, we have a proof that it helps the language, uh, language model, pre-trained language models, uh, without any fine-tuning to better utilize long context. And we, uh, provide a, a few benchmarks. Uh, as you can see, for example, uh, smaller models have a better, um, mmm, uh, more, uh, more improvement. So here we have a, for example, long mo, uh, long context, so the generic pattern that we use for long context, so just the context and the question, whatever the, uh, the benchmark is. Then we have RAG, which means basically that we, um, we extract, uh, instead of, um, giving the entire context plus the margins and then the question, we are basically extracting each of the chunks separately, asking the language model which one of them is relevant, and then only fill, uh, providing the relevant ones, which is what we would do in RAG. Um, and then, uh, asking the language model to leverage only the relevant ones at the end or the Writing in the Margins approach, which is all the context plus the margins that were extracted during this, uh, Chunked Prefill and then, uh, the question at the end. Uh, now, how is this different from a prompting strategy? Because, uh, you may be thinking, "Okay, but we can already take a very large prompt, se- split it into chunks, and then, um, use the language model to kind of summarize each of this chunk independently, and then send it to the language model again to answer the question." ### Cost **Umar Jamil** [15:43] Well, it's a matter of, uh, cost. So let's talk about cost. Imagine that we want... we have this very big book made up of 10 chapters, and then we have a question at the end that we want to get an answer to, which is, what is the answer to life, the universe, and everything? Now, what, uh, Writing in the Margins would do is, would, uh, prefill the, mm... If you don't use Writing in the Margins, what you would do is you would feed the first chapter of the book and then extract some margin with that. Actually, let me show you this other slide. Then you would take the second chapter of the book and then extract the margin with that, then the third chapter of the book and extract the margin with that, et cetera, et cetera. Suppose that each of this chapter is one hundred thousand tokens. Then the cost to generate the margins, and suppose that the margins are really small, it's more or less one million tokens, because one hundred thousand tokens multiplied by 10 chapters is around one million tokens. But then, then you need to also send the entire book plus these extracted margins again to the language model to, um, to, to generate the answer, and it would cost you another million because the model has to reprocess th- this prefilling again of one million tokens. So it would cost you two million tokens. But with Writing in the Margins, it would cost you a, for one million prompt, more or less it would cost you one million tokens, because you don't have to re-prefill the, uh, the, the entire context because you are, you have already prefilled it. So the cl- the KV cache is growing and you're extracting some information, and then you don't have to repopulate it, which is what you would do if you, uh, uh, treat each, um, chunk, uh, independently and then, uh, do kind of like a, um, let's say the chunking that we commonly do. Um, so what are the advantages? Well, it's compatible with any transformer model. Um, there are other advantages that I want to show you in a video that I made. So let me show... This is on my LinkedIn post. ### Demo & Classification **Umar Jamil** [17:45] Uh, I don't know if you can see now again my s- my screen, right? It's a few- **Vibhu** [17:49] Yeah. Looks good. We see the LinkedIn video. **Umar Jamil** [17:51] Perfect. **Vibhu** [17:51] Yep. **Umar Jamil** [17:52] Okay. So we have a large document, suppose one million tokens, and then we have a question, like how many employees were hired in 2006? Um, now, when we have a very large prompt, it must be prefilled in the KV cache by chunks, and this operation is called the Chunked Prefill. So what happens with Chunked Prefill is that you take the first chunk and you prefill it in the KV cache. Then what we do is we add to the KV cache this extractive summary prompt, which is, for example, "Use the text above to extract information about the following query: How many employees were hired in 2006?" And then we generate tokens using whatever is inside of the KV cache, which is the first chunk plus this instructive, uh, extractive sum, uh, prompt. And suppose that we generate a few tokens, which are visible now. We take these tokens, we save them, so we decode whatever is generated. We're using the tokenizer, and we save it, and then we remove it from the KV cache. Now, if you are looking from a implementation-wise point of view, you don't actually remove from the KV cache. You just... Usually, the KV cache allocation is a static, but even if you use vLLM, it's used, it's done using the, the so-called like paged attention. So you actually a-allocate pages of KV cache actually. Um, so, uh, basically, we, we, we, we... It's not like you are, uh, removing stuff from the KV cache. You are just resizing the tensor, which is actually, um, O (1) operation, so it, it doesn't have a-additional cost. You just keep track of how many tokens there are. Um, so anyway, we, we take this margin, we save it somewhere, and then we prefill the second chunk to the KV cache. And then we again a-add another extractive summary prompt, and then we leverage it to generate the second margin, which will depend on the first and the second chunk of the, um, of the prompt, et cetera, et cetera, for all the chunks. So we will have a list of margins. And then we can also classify these margins. So we can also say, uh, which... Because some of the margins may, uh, like, be, uh, the model ha-hallucinating or the model just saying, "I cannot find this information," or et cetera, so we can also classify them. We can either use an auxiliary classifier, or we can use the model itself to classify them. And we show in the paper that you don't have to actually create a new request to the model to classify these margins. You can actually overlap the generation of the margins with the classification of the previous margin using the same request in the batch. Uh, does the margin pertain to all previous chunks or only the current one? Uh, all the m- uh, chunks up to that margin. So the first margin, only the first chunk. The second margin, the chunk one and two. The third margin, chunk one, two, and three, et cetera, because we want to leverage the, uh, uh, prefilled KV cache. Um, okay, so we classify these margins, and then we append them at the end. And then we append the question. And then we generate the answer, the final answer. So, uh, the advantage is that, as I said before, we exploit chunked prefill of a large prompt to generate intermediate summary, so we are, we are, we are exploiting something that we are already forced to do. But, uh, we are not leveraging this right now. Uh, so it's kind of comes for free just with a minor, let's say, uh, compute because, uh, you need... You have the com- the, the, the, the, the cost of generating these margins. But you avoid the bigger cost of prefilling. So if you just use chunks tech- uh, chunking techniques like you can do with the LangChain, for example, um, you pay twice the cost of prefilling. If you do this system, you don't pay twice the cost of prefilling, which is very expensive. Um, and to give you an insight on how expensive is prefilling, basically, um, in most, uh, cases, so whenever you work with the OpenAI or with Cohere or any other provider, whenever you send a request, your request is always overlapped with the token generation of other requests. So the first time your prompt comes to their server, they are overlapping the prefill of your request with the token generation of others because the prefilling is compute, uh, bound because it's very e-expensive computationally, while the token generation is memory bound. So to, to, to always utilize the GPU fully, uh, they over- uh, they kind of, um, schedule together one prefill with multiple token generations. So, um, so it's compatible with any off-the-shelf language model without any fine-tuning, and we saw, we show some, uh, uh, benchmarks in the paper. Uh, it improves the ability of any language model to extract relevant information, so solving the lost in the middle problem. And another, um, cool, um, uh, thing that you can do is basically, m- because now you generate these margins, uh, while prefilling the, the, the, the prompt, you can also feed back this margin to the user, and the user can classify them for you, like a thumbs up or thumbs down. So it ca- adds a human in the loop. Um, and also the user can visualize the progress of how the prefilling is going. Because when you have a very, very large prompt, uh, I believe that because the cost of prefilling is quadratic, it will become really, um, it will, it will become really, um, expensive to prefill it, and the user may have to wait many seconds. So you can actually give a feedback to the user of how con- how much context has been processed, and you can actually leverage the waiting time of the user to give you thumbs up or thumbs down on these, on these margins, which can actually improve the ability of the language model to, uh, use them. And the user can also early exit. So the user found the relevant information in one of these margins. The user can say, "Okay, stop inference," and the user would not have to pay for, you know, all the context, uh, being processed. And we also provide an implementation. So if you go to this, uh, URL, so github.com/triter/writing-in-the-margins, you can find our implementation on how we actually do this stuff with the KV cache. I don't know how to delete this line. ### Code **Umar Jamil** [24:21] It's so annoying. Uh, let me check. I believe it's annotate and then delete, clear. Clear all drawings. Okay. So if you go here, you can see, uh, what we do. Basically, um, it's a simple im- im... It, it works with any language model. We here, we provide a, a demo, uh, with the Llama, Phi, and Qwen. Um- Uh, here we show, for example, um, this, this code that is present here in the GitHub repository matches exactly the code that we present in the paper, which is the pseudo code that you can see here. So, uh, how we split into segments and, uh, how we prefill into the KV cache, and how we delete stuff from the KV cache. So it's present here. So here you have a very simple, like, uh, we also show the state of the KV cache at each, uh, line of code so that the user can understand what is happening, um, here. And this is the code for the method that we use to delete stuff from the KV cache. All right. Uh, let me see if there is something that is missing here. Yeah. Here we provide a comparison of, uh, how it differs from RAG, and, uh, how it differs from just long context processing. Questions? ### Q&A **Umar Jamil** [25:41] All right. Let me check. **Guest 2** [25:42] I don't see you in the chat, but Eugene is, like, crushing it answering qu- answering the questions in the chat. Thanks, Eugene. **Umar Jamil** [25:49] All right. **Eugene** [25:50] My pleasure. Thank you for taking time to share with us about, uh, the paper, and even preparing slides. **Umar Jamil** [25:56] Uh, yeah. This, uh... Okay. The, the, the slides were from another talk I gave, uh, in, in the company, so it's, uh, reusing stuff. But yeah. Thank you. Thank you, everyone. Um, so let me go through the questions, if there is something in the chat that I can answer. Um, uh, yes. This is not any... The, we are not doing any change to the model architecture, so you don't have to, uh, fine-tune anything. You don't have to change anything. Like the, um, uh, can you use this stuff with, like, a LangChain? No, because it requires a modification on how the inference engine is using the model. So it, uh... Because when you work at the KV cache level, you cannot just, uh, uh, work with the APIs and tell them to, you know, remove stuff from the KV cache or overlap stuff in the KV cache. But it doesn't require changes to the model, to the weights of the model. That's why we talk about no fine-tuning here. Is the extractive summary prompt just instruction to produce the margin? Yes. So the extractive summary prompt is basically, uh, a prompt that we add after each chunk to extract reli- relevant information about that query. So it's not just to find the relevant information, but about the specific query, because we, this inference pattern that we introduced is specific, uh, for those prompts that are composed of a context plus an instruction. So we always know what is the instruction. That's why, um, this is, I mean, the, the best use of this, uh, this inference pattern. Now, in the, in the paper, we also show, um, how Chunked Prefill works at the KV cache level. So if you are familiar with how the KV cache, with the queries, the keys, et cetera. Uh, but we also show how to overlap the computation of the margin with the classification of a margin. Uh, and this is exactly what actually the, the representation of the KV, uh, cache, uh, during the prefilling of one chunk, and how it can be overlapped with the classification using the same language model and the same request in the same language model. Um- **Guest 2** [28:09] S- sorry, could you go deeper into overlap? I, I, I don't know where, like, overlap happens. Is it between the different chunks? Are you overlapping the different chunks with the margin? **Umar Jamil** [28:18] Um, let's talk about overlap. So I am talking about, uh... Let's say, first visualize it. Uh, let's say here. We do have a nice representation of that. So it is here. So you extract the margin, and you need to find a way to classify it. You can either use an auxiliary classifier, so use another model to classify it as relevant or irrelevant, or you use the same language model to classify. But if you want to use the same language model to classify, you would need to create another request in the batch, because you don't want the, uh, classification request to visualize anything in the KV cache. You just want to ask the language model, "Okay, the... I ask a language model to extract information about, uh, this query here. So is Ethan Washington in a marble floored room?" And the language model, uh, extracted this stuff here. Is it relevant to the query or not? If you want to do it, you would need to create another request in the batch. But we show here that you can actually, um, in the Chunked Prefilling... So let's go here, Chunked Prefilling. In the same request in the batch, so when you do Chunked Prefilling, basically what you are doing is you are adding the first chunk to the KV cache. So the, the, the keys and the queries are the first chunk. So this is C1 that you see here. And then what do we do is we actually add after this, we also want to add an extractive summary prompt, right? Uh, and then we use this one to generate tokens. So now the token generation has, um, is, uh, how to say? This is the part of the prefilling of the first chunk, so the first chunk plus the extractive summary, and then we use it to generate tokens. So this is the first, uh, margin token generated. Usually, we, uh, pre-allocate the KV cache. So the KV cache is not, like, growing tensor. We pre-allocate it with a fixed number of, let's say, padding tokens, but they are not really padding tokens. They are just unused spaces in the KV cache. And then we replace them with the tokens that are actually generated from the language model. So suppose that you have these, uh, unused tokens, which I call padding here. Basically, what do we do? After we have generated the first margin, we delete this margin, right? And also the instructive token. So what we are actually doing is we don't delete anything. We just change the pointer position of the KV cache on how many tokens are used. So now the pointer, suppose it's pointing here. Then we can prefill the second chunk. So the second chunk needs to attend to all its tokens in, um, a causal way. So each token in the second chunk needs to attend to only itself and all the previous tokens in the same chunk, but also needs to attend to all the past tokens of the first chunk that was already prefilled. And we also need to prefill the instructive summary prompt, which can visualize all the past tokens that it has seen. But while, uh... Then, then, um, we can also, uh, skipping some tokens that we reserve for the generation of the margin, we can al- also prefill the classification of the, uh, the, mm, the classification instruction for the first margin, which was generated before in the previous step. And then during the token generation step, so this is the prefilling along with the instructives, uh, the pr- the prefilling of the second, uh, segment. After we have prefilled the second segment, along with the first generated margin, we can generate the tokens of the second margin, but classify the first one, which we already obtained in the step before. So we are generating two token sets here in the token generation step in the same request now. One, using only the part relevant to the first chunk, the second chunk and the extractive summary of the f- the, the, the, of the, after the second chunk. And one, using only... As you can see, this is the attention mask, right? And one is only using the, um, the part that is relevant to classifying the first margin that was, um, instructed in the previous step. So you can do it also like this. **Eugene** [32:27] Thank you, Umar. **Umar Jamil** [32:29] Yeah. **Eugene** [32:30] Uh, sorry, there's a question in the chat. Um, and I think from your explanation, I think the answer is clear. So when you are creating the margin for the second chunk- **Umar Jamil** [32:40] Mm-hmm **Eugene** [32:40] ... you're actually paying attention to the first and second. **Umar Jamil** [32:45] Yeah. Okay. So you can change the, uh, attention mask to only look at the latest chunk. However- **Eugene** [32:50] Yep, that's exactly the question. Yeah. **Umar Jamil** [32:53] Yes, but it's, uh, not possible, actually. I mean, uh, let me clarify why it's not possible because the KV cache is made up of contextualized tokens. It's actually these tokens are not single, they are contextualized. So the pos- the token number one in the KV cache is a contextualized version of the token number zero and one. The token number two in the KV cache is a contextualized version of the token zero, one and two. So if you tell the model to only look at the last tokens, uh, you are creating an autoregressive model that is, uh, generating the logits of p of, let's say, x ten, but only looking at p of x nine, x eight, which are contextualized token con- that contain information about seven, six, five, but you are not using them, so you are actually ch- going out of distribution. So this is why you- **Eugene** [33:42] I see. **Umar Jamil** [33:42] Yeah. **Eugene** [33:42] Thank you. **Umar Jamil** [33:48] Uh, how much of the KV cache do you prefill with the chunk versus leave it? Um, you can, you can actually, uh... Or well, okay, if you use, for example, vLLM, they use this called, thing called the pages attention. So actually they prefill, uh, they allocate one entire page, which is actually a lot of tokens. So it's like another chunk, which is more than enough to generate the margin. So what are the next steps for this? Well, the next step is, uh, for sure we are sending it to, uh, conferences, uh, get it published and, um, presenting it around. Uh, but we are recently focused on long, uh, context modeling. And actually we are, um, uh, looking at, you know, how long context modeling, uh, long context can be better leveraged. So we will be working in this field, actually. Uh, we will be, um, uh, how to say, um, we'll be researching a lot in this, uh, field. **Eugene** [34:54] I think there's a question from Ahmad. How are the queries chosen for the query-based summarization? **Umar Jamil** [35:01] Uh- **Eugene** [35:02] I think it's a classification, right? **Umar Jamil** [35:06] Uh, yes. Okay. So the queries, basically, uh, we work with a, a prompt that is made up of context plus query. So we al- always know what is the query. That's the structure of the prompt that we, uh, work with. Uh, what's the use case at Writer that led you to this research? Well, we are... I am personally very interested in long-context modeling, and I am given the freedom to research, uh, what I like. And, uh, Writer is also interested in long-context modeling, so things intersect and this, here we are. And then, you know, we have our many smart people working together. We did a few brainstorming and, uh, yeah. Uh, what's the latency you see for typical request? Uh, well, you are... The la- there is no kind of latency increase because of this. You are just paying more price to generate more tokens in intermediate cases. Of course, what would happen is that, uh, you have, uh, before, for example, you need to, um, process the entire prompt at once, so chunked prefilling, chunk prefilling, chunk prefilling. Now you have chunk prefilling with some token generation, which will slow down the entire request. But, uh, you are actually getting something back, which is feedback, and you are getting the possibility to see what the model is actually seeing at each step. So you get human in the loop, so the human is waiting, but is waiting, uh, let's say with some feedback, which is nice to have progress bars, right? **Eugene** [36:41] You, and maybe this is sensitive, do you happen to have a demo showing how this actually looks like in the user interface? Or is it something that we have to sign up to Writer to actually see? **Umar Jamil** [36:50] Uh- We don't have that, but we are working on demos, yeah. Uh, here we have, you know, we have a concept on how it would look like. Yeah. **Eugene** [37:04] Thank you. **Umar Jamil** [37:08] Uh, so okay. In some, uh, cases, uh, the writing in the margin does not wor- uh, work well as other methods. There are two, uh, factors. First of all, uh, because we are... each margin is kind of an ex- a summarization of what is present in the context. It depends highly on how good that model is at summarizing. So the better the margin, the better the information it will extract, and the better it can be leveraged. So if you think about the student, if your note-taking skills are not so good, then probably your notes will not be useful. Uh, the second thing is actually the comparison here that you see with RAG. This RAG, actually, w-we put ourself in the worst condition possible, which is let's help RAG beat us. But then actually RAG doesn't beat us. How? Usually in RAG, what you do is you have these chunks and you, uh, uh, extract some, uh, vectors of these chunks, and then you match them with, uh, um, dot product or whatever with the query. What we did with, uh, RAG actually is we asked the language model to see if the, uh, RAG, yes, it was charitable. Because we asked the language model actually to, to, to see if that particular chunk is relevant. So actually, you have a 70 billion model telling you if that chunk is relevant compared to extract some vector and map it with dot product. I mean, we helped RAG a lot. So actually, if we did actually a RAG approach, like a naive RAG approach, we would do much better. **Eugene** [38:43] Thanks, Umar. Do we have anyone else have questions or wanna come on screen to just ask Umar and Sam more questions? ### Efficiency **Guest 3** [39:11] If, if there's nobody else that is interested, I actually am having a little bit of trouble wrapping my mind around why chunk prefilling is so much more efficient. I, I, I looked at the, the sort of, uh, the reference and I, I kinda get the idea, but maybe you can help me understand the intuition. **Umar Jamil** [39:32] Okay. Uh, it is not, uh... First of all, chunked prefill doesn't exist because it's more efficient. It's because we need it. We must do it. **Guest 3** [39:40] Okay. **Umar Jamil** [39:41] So when you prefill, uh, a, a chunk into the language model, let, let me show you actually. Here we have the KV cache presentation, right? So when you prefill a chunk in the language model, let's say this one, chunk number one, C1, you are generating a quadratic, uh, um, matrix, as you can see. Uh, if you have four tokens, you are generating a four-by-four, uh, matrix, uh, which is prohibitive to generate for very long prompts. Like you- **Guest 3** [40:07] Yeah **Umar Jamil** [40:07] ... imagine you have one million tokens, that's gonna be one million by one million matrix, where each of these values is actually a dot product of a vector, and then, uh, the computation cost of that also. It would really be very slow, and the GPUs are really good at parallelizing. In this case, if when you have a lot of operations, they will actually be serializing **Guest 3** [40:27] Yeah **Umar Jamil** [40:27] So you are, we are forced to do this chunk prefilling. Then we are doing, okay, we do this chunked prefilling, so chunk one, chunk two, chunk three, chunk... But we are not leveraging these chunks that we are... because we are forced to do it, right? So it's slower than just doing it in one pass. But since we are already forced to do it, why not use them? **Guest 3** [40:54] Yeah. Yeah. No, I, I, I, I definitely, I think I got mo-most of the paper, just the chunk prefilling part, the, the background I don't quite understand. **Umar Jamil** [41:02] If you want, uh, like, uh, more information on chunked prefilling, I can, like, give you some, uh, references. One is the vLLM, uh, page. They are actually, um, in the vLLM now. They are, uh... it's an experimental implementation. Uh, there was a NVIDIA, uh, explanation on chunked prefilling, so I will send a, uh, link later. Uh, NVIDIA published recently an article about chunked prefilling. But basically, prefilling is, uh, the most expensive part of, uh, working with long prompts for language models. That's why, yeah, they need to- **Guest 3** [41:34] So but why... I, I guess I was having trouble understanding why, why that is. Is it just because it's quadratic and so you have to break it up into chunks? Is that the, the- **Eugene** [41:43] Maybe, maybe I can take a stab at it. So you can imagine- **Umar Jamil** [41:47] Okay. So, uh, let me look at attention mask here. Here I'm generating the first margin. **Eugene** [41:51] Yeah. **Umar Jamil** [41:51] How many dot products are, am I doing here? I am doing, uh, I have already seven tokens in the, uh, KV cache, and I am generating the eighth token, so I am doing seven dot products. So token generation, which means generating one token using whatever is in the KV cache, is linear with respect to whatever is in the side of the KV cache. Prefilling the KV cache is quadratic, and mostly because it's quadratic, it's very expensive. So we are talking about something that is linear with quadratic, so. And if you consider about prompts, long context, if you are working with a two million context window, one million and nine hundred and ninety-nine thousand will be prompt. Nobody will ever generate more than, let's say, five thousand tokens. So- **Guest 3** [42:34] Yeah **Umar Jamil** [42:34] ... the biggest part, the most expensive is actually prefilling. **Guest 3** [42:37] So, so, so just to get an intuition here, the, the other extreme is that you, your chunk size is one token, right? So what's the trade-off? **Umar Jamil** [42:48] So what they do is basically they try to put as bigger as po- as big as possible un- until it fits in the GPU. So they usually, um, suppose, uh, I think, uh, good, uh, numbers are, like, 4,000 tokens or 8,000 tokens or s- something in this range. And, uh, as I said before, usually, uh, the token generation is, uh, memory bound, means that the limitation is only given by how much your KV cache can hold, so the memory can hold in terms of KV cache, while prefetching is compute bound, so to maximize the U- GPU's utilization whenever you work with OpenAI or Cohere or what, they just overlap your new request with other people's old requests. So while they are generating tokens, they are also prefetch, so the GPU is utilized 100%. **Vibhu** [43:34] Okay. Yeah, no, that, that's helpful. And I... If you do send those, uh, links, I'll definitely read them. Thank you. **Umar Jamil** [43:40] Okay. **Guest 2** [43:41] Is there, is there a break-even point where at, at, at, like, a certain context length it becomes more valuable to do writing with margins than, um, th- than just using the LLM by itself? Like, it... Where, where the, where the compute equals out? **Umar Jamil** [43:59] I believe- **Guest 2** [43:59] Or is it altogether margins is better? **Umar Jamil** [44:02] I believe, okay, writing in the margins, it's like, uh, you read a book and you have margins versus reading the book. I think it's always convenient to read the book with the margins, because you are actually paying the price for that, right? We are, you... It's not something that comes for free. You are actually paying the cost of generating this margin. So you are actually putting there some effort, and then you, you leverage it. Then we, what we could do is, okay, does it always help? So far, yes. So it's not like something that you get for free, right? So you pay and it's... Is it worth it? So far, yes. And if it's always worth it, I, so far, from our data, it's always worth it. Like, it doesn't- **Guest 2** [44:45] Really? Like, even if, even if you're literally- **Umar Jamil** [44:48] Maybe if you have long- **Guest 2** [44:48] ... just talking about a chunk. If you have, like, a sentence. **Umar Jamil** [44:52] Yeah. **Guest 2** [44:53] Instead of a book. Oh, really? That's crazy. **Umar Jamil** [44:55] Then it's not, yeah, then it's not convenient because, uh, in that case you are not even doing chunk prefetching, right? Because if you have only small context, then they just prefetch it at once. **Guest 2** [45:05] Okay. **Umar Jamil** [45:05] But when you have long- **Guest 2** [45:06] Yeah. **Umar Jamil** [45:07] Yeah. **Guest 2** [45:08] Makes sense. **Vibhu** [45:08] Can I- **Guest 2** [45:08] Once you start growing in chunks **Vibhu** [45:11] ... I think the topic still stands, right? Like, let's say you've got a paragraph and your chunks are sentences, or you've got a page, right? So you've got 1,000 tokens, and your chunks per se is every sentence relevant. So some form of highlighting is, you know, you have a one page, but every sentence you highlight what's relevant or not. At that level, it's, it's kind of negligible to throw the whole thing in a prompt versus do I wanna highlight 7 of the 40 sentences and do this approach? I think that's the, that's the other non-extreme, right? **Umar Jamil** [45:44] Yeah. So basically whenever the, the context can just be prefilled in the KV cache without any chunked prefetching, I believe it's not worth it to use it. But if it's long, then it helps. And it helps much more than, uh, chunking separately like we do with the APIs, right? With the LangChain. Because you are paying double, twice the cost of prefetching. In this case, we are only paying once, once the... And we also prove in the ablation studies that actually it's always convenient to send the context plus the margins, never just the margins. So you can see here, the, this, uh, ablation cont- uh, compression. So if you only send the margins or only the context, it's always worse than providing them both. **Vibhu** [46:36] Context being the whole, right? **Umar Jamil** [46:39] Uh, yeah, the, the entire book. **Vibhu** [46:45] A- and so build- building off that ablation, l- let's say you've got a model that doesn't have the context and you have to do some sort of chunking splitting. Uh, let- let's say, you know, you have a total context of 8,000 tokens and you have a million token document, there's approaches of, you know, how you can process chunk by chunk and then combine. So would the, you know... What you would expect is you could, for each chunk, do this writing with margins approach at every level and then scale that down with how many ever steps you need. Is that intuition still pretty accurate? **Umar Jamil** [47:18] Uh, I believe with 8,000, um, with 8, uh, thousand, uh, how to say? Context window, uh, I believe that the latency would be higher, right? Because you, at each step you are adding more. Yes, at that kind of latency, it's even more convenient to just, uh, do, uh, independent chunking and generate in that kind of lane. Because if you, you can always split the, the, the, the context into chunks, and then you just send multiple requests, and you can pay the price of prefetching in that kind of range. But when you are talking about 64,000 tokens, that start making more sense to use this approach. So for that level, I think it's all the traditional approaches, uh, they work fine. **Vibhu** [48:09] Any other questions? **Umar Jamil** [48:12] Uh, well, I think, uh, another question that came out, uh, was why now, why, right, I mean, why nobody thought about this before? Because actually, uh, we didn't have long context, very long context, um, uh, models before, and we were not forced to even do chunked prefetching. ### Future **Umar Jamil** [48:30] So as you, as you can see from vLLM, they have this, a feature is an experimental feature right now in vLLM. So it's because right now we need this chunked prefetching, and everyone is doing it, so that's why we have this. So it's always, you know, uh, innovation is always starts from some problem that you face and some need that you have. So right now we have this need and we have the capability, so r- that's w- how we came up with this. **Vibhu** [49:07] Awesome. We have a few more minutes. If anyone else has last-minute questions, um, please feel free to ask. And big shout-out, thanks to the Writer team for presenting. **Umar Jamil** [49:20] Thank you guys for, for listening. Um, you are welcome to, um, send us, uh, your questions. Uh, we have a, you know, GitHub, uh, repository. I, I, I, I suggest w- watching the code. It's really, you know, very commented, and it follows the same, uh, kind of pattern that we shared in the paper, so it's easily understandable for everyone. And we ha- we did a lot of nice tricks, you know. Like, one of the tricks is, like, you can always delete stuff from the KV- from the end of the KV cache, and you also know why now. **Vibhu** [49:54] It's a interesting project if anyone's interested. On Fridays, we have a similar AI in Action section where we try to take practical outside of paper. There's the code up, there's the paper up. If anyone wants to run it and present it, share their learnings, it'll be a really good learning exercise. But that's always there. Um, I guess we got a question from Jimmy. Any future work in this direction? **Umar Jamil** [50:17] Any future work in this direction? Well, for sure we will keep working on long context and how we can better level at long context. So there is another kind of problem with the long context, which is, uh, uh, the, the, the how language models use long context actually depends highly on this attention mechanism and how the softmax works. So we have seen with the paper called Sink Attentions that actually the language model allocates a lot of, uh, um, lot of... Because when you do, uh, the attention mechanism, you are doing a weighted sum over the tokens, and each token is given a weight, and we see that most of the weight is given to the first few tokens. So there is a lot of, uh, research in this area. Recently I saw, few days ago, another paper came out called the Sigmoid Attention, which is also studying, you know, the distribution of these logits, uh, in the talk. So I think, uh, the, the, the attention mechanism will play a big part in how we can extend the long context. So if we can also fix this part here. So I am, am very interested, you know, in the KV cache and, uh, optimizing, uh, long context modeling. So, mm, we are, we are working in this direction. Because it's, it's needed by the market, and, uh, also it's... I like it. And, uh, I, I, I think it's cool to be able to analyze an entire book or an entire code base instead of hoping that RAG finds the right one. **Vibhu** [51:45] Awesome. Well, uh, big shout-out to Writer team. Always great to have you guys present. Um, Sam is in Discord as well. I'm sure he'll relay questions and stuff. We've got the recording. We'll share it with your team. I don't know what you choose to do with it. ### Outro **Vibhu** [52:00] But, um, next week we've got Swix. He'll be presenting some of the Strawberry, Q*, QuietStar, all those papers. So he'll be doing that next week. And then the following week, if anyone's interested in anything, volunteers are always open. I posted a few papers in, um, Paper Club. I think there's also the Mistral stuff, so if anyone wants to lead, pop in there. Otherwise, um, next week Swix is doing Strawberry and Star stuff. So that's on the agenda. **Umar Jamil** [52:30] Good. **Vibhu** [52:31] Cool. Thank you guys. Thanks everyone. **Umar Jamil** [52:34] Take care everyone. **Vibhu** [52:36] Take care. **Guest 4** [52:38] Thank you. **Guest 2** [52:41] Vibhu. Vibhu. **Vibhu** [52:43] Yes. I was just about to end the meeting, yes. **Guest 2** [52:45] Vibhu, question. Is there any- you can... Um, is there any way you can copy over the comments? **Vibhu** [52:54] Um- **Guest 2** [52:54] Um, I'm, I'm trying to do it, but it seems to, like, lazy load. Like, as you scroll up and down, it's really painful. **Vibhu** [53:01] Let me see if I can extract these comments. Do you know if they normally get saved as a Zoom recording? **Guest 2** [53:12] I'm, I'm using- **Vibhu** [53:14] You can click Save Chat. Where is Save Chat? Anyone wanna help me out? **Guest 2** [53:17] I might be able- **Vibhu** [53:17] Chat. There's a- I was able to copy the comments without any problem. **Guest** [53:23] Uh, the file is usually saved on the host computer in a folder like Documents, Zoom, meeting date and time, if you're on Windows. See there's a chat log file that, uh, as long as the, we stay recording. **Vibhu** [53:36] Okay. Uh, I got it. I just, I just saved the chat. I'll, I'll throw it in Discord. I have a text file of it. **Guest** [53:41] Hell yeah. Yeah, slides would be great too if we can grab them. Uh- **Vibhu** [53:44] Uh- **Guest** [53:44] ... yeah, get it all. **Vibhu** [53:45] I'll ping 'em. I'll ping them right now. **Guest** [53:48] Sweet. Perfect. **Vibhu** [53:49] Perfect. Thanks guys. **Guest 2** [53:50] Sweet. **Guest** [53:50] All right, Gigi. **Guest 2** [53:50] All right, thanks. Bye. **Vibhu** [53:52] See ya. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.