# RWKV: Reinventing RNNs for the Transformer Era Latent Space · 2023-08-31 Eugene, CTO of UILicious, introduces RWKV, a recurrent neural network that matches transformer-level performance while scaling linearly with context length, solving the quadratic cost of attention. He explains how RWKV uses Attention-Free Transformer layers and parallelizable training, achieving competitive results on 7B and 14B models. The community-driven project prioritizes multilingual support, with a tokenizer that handles languages without spaces. Eugene recounts his journey from GPU.js to finding RWKV, and highlights use cases like long HTML analysis where transformers fail. He also discusses the token crisis, diffusion models for text, and how the AI waifu community drives optimization and alignment research. Finally, he advises AI engineers to focus on practical prompting and data curation, noting that deep architecture knowledge is optional. ## Questions this episode answers ### What is RWKV and how does it work without attention? RWKV is a modern recurrent neural network that matches transformer LLMs without attention. Eugene explains it uses "weighted receptors" from Apple's Attention Free Transformer, replacing multi-head attention with a linear-complexity "time mix" and "channel mix" mechanism. It can be trained in parallel like a transformer, but inference scales O(1) per token, avoiding quadratic costs. Benchmarks show it matches GPT-NeoX performance on the same data and parameter count (14B). [31:46](https://addtry.com/2d99491b-3bec-4860-990c-61c5192442c7?t=1906000) ### How does RWKV's linear scaling compare to transformers' quadratic scaling for long contexts? RWKV processes each token with O(1) operations and maintains a state, so generating N tokens costs O(N) time. In contrast, transformers recompute attention over all previous tokens for each new token, scaling quadratically O(N^2). Eugene illustrated this with a chart showing cumulative GPU time: RWKV's linear curve stays flat while the transformer's rises sharply. This makes RWKV far cheaper for long contexts, both in training and inference. [1:02:32](https://addtry.com/2d99491b-3bec-4860-990c-61c5192442c7?t=3752000) ### Why hasn't RWKV become as popular as transformer models? Eugene attributes RWKV's lower profile to poor marketing, a late paper, and the community's focus on supporting non-English languages rather than optimizing for English benchmarks. Early models used smaller datasets (Pile) and thus underperformed models like Falcon on standard evals. With the new "World" model and plans to train on larger English datasets like RedPajama, they aim to close the gap. [1:07:32](https://addtry.com/2d99491b-3bec-4860-990c-61c5192442c7?t=4052000) ## Key moments - **[0:00] Intro** - [0:49] GPU.js originally compiled JavaScript to WebGL shaders to run matrix multiplications on the GPU for fun. - [1:53] GPU.js outperformed V8 on matrix multiplications beyond 2,000 dimensions but required data transfer to the GPU. - [4:21] GPU.js parses JavaScript AST and translates it to WebGL shader code, simulating missing types like big ints. - [5:15] Brain.js and TensorFlow.js allow teaching neural networks in the browser without CUDA, even if only for toy models. - [7:00] UILicious simplified browser automation by making the API synchronous and text-based, avoiding async and CSS selectors. - **[7:54] Alternatives** - [9:36] Post-transformer, fine-tuning needs only 1,000-10,000 data points, compared to pre-transformer requiring 100GB-1TB of data. - [10:53] Q: Did UILicious build its AI on GPT-3? - [12:12] Salesforce CodeGen was trained only on MIT-licensed code, avoiding problematic licenses like those possibly used by GitHub Copilot. - [14:00] Eugene's interest in transformer alternatives began when customers asked AI to analyze test failures on large web pages. - [15:33] Their AI crashed on Amazon.com because the HTML exceeded 5MB, blowing past token limits even after stripping CSS and JavaScript. - [16:20] Tokenizers handle HTML poorly; a single complex page can expand to over a million tokens, far exceeding typical context windows. - **[20:05] Discovery** - [20:09] Eugene found RWKV while searching for non-English-friendly models because most open-source models degrade when adding other languages. - [23:28] Most open-source models exclude non-English data from training because it hurts benchmark scores at small parameter counts. - [26:13] RWKV started with English, Chinese, Japanese, and Korean because volunteers from those communities contributed datasets and feedback. - [29:19] Community hackers contribute for personal enjoyment, wanting an AI that works in their language rather than chasing leaderboards. - [31:01] Q: Is the AI Husbando market larger than the AI Waifu market? - **[31:46] RWKV Overview** - [32:25] RWKV matches GPT-NeoX performance without attention layers, using a recurrent neural network that can be trained like a transformer. - [34:02] RWKV uses the Attention Free Transformer (AFT) from an Apple paper, replacing multi-head attention with weighted retention states. - [37:46] Q: What does RWKV stand for? - [38:19] RWKV has 7B and 14B base models, plus instruction-tuned Raven models and a 100-language World model now in training. - [39:46] Raven instruction data was scrubbed of 'as a large language model' phrases to create uncensored models. - [41:58] The World model training data is 70% English, 15% multi-language, 15% code; multi-language percentage was increased to support more users. - **[56:12] Architecture** - [56:31] RWKV can be rolled out as a transformer during training by splitting states across layers, enabling parallel computation. - [59:07] Q: What is the Big O complexity of RWKV inference? - [1:00:42] RWKV's Time Mix retains long-term memory with trainable weights, while Channel Mix handles short-term memory that gradually fades. - [1:02:47] RWKV's linear scaling vs transformer quadratic scaling results in drastically lower cumulative GPU time for long sequence generation. - [1:05:24] RWKV reduces working memory during inference, allowing token-by-token processing with minimal state history, enabling runs on Raspberry Pi. - [1:07:08] RWKV performance is competitive or better on reasoning benchmarks, showing no trade-off between speed and quality. - **[1:07:32] Obscurity** - [1:08:10] Q: Why isn't RWKV a bigger deal despite its technical advantages? - [1:10:35] RWKV's impact is larger in non-English communities because it prioritized multi-language data from the start. - [1:12:09] Eugene predicts RWKV will scale to 40B-60B parameters and match larger transformer models when trained on datasets like RedPajama. - **[1:12:32] Scaling** - [1:15:21] The goal is to extend RWKV's non-lossy context window to 100K tokens to process entire complex web pages without losing memory. - **[1:16:32] Community** - [1:16:33] The all-volunteer RWKV Discord community self-organizes by area of focus, with Blink training and individuals contributing where they have incentives. - [1:22:28] Blink wants RWKV to become the Linux Foundation equivalent for AI, a neutral, open-source, non-commercial foundation. - [1:25:31] Eugene speculates diffusion models might solve the token crisis by enabling multi-epoch training without overfitting via noise randomization. - **[1:25:32] Diffusion** - [1:27:49] Llama 2's training on 2 trillion tokens did not plateau, but high-quality text tokens are becoming scarce—the 'token crisis' is real. - [1:29:38] Diffusion models can train for 200+ epochs because added noise makes each epoch effectively a different dataset, preventing overfitting. - **[1:32:32] Advice** - [1:32:50] AI engineers don't need to know math or model internals; they should focus on using, prompting, and chaining LLMs for user applications. - [1:35:20] Training and fine-tuning lack fixed rules; each domain like music may require unique hyperparameters found through trial and error. - [1:39:19] Q: Which resource does Eugene recommend for understanding backpropagation and model internals? - [1:42:20] Non-AI software engineers can optimize AI models with techniques like memory mapping, shader alignment, and batch size tuning. - **[1:45:00] Mind Upload** - [1:45:37] AI Waifu community members are highly motivated and technically competent, driving optimizations from prompt chaining to inference speed. - [1:47:05] Character AI alignment means staying in character, even if it involves making intentional mistakes, not adhering to ethical guidelines. - [1:53:16] Eugene predicts the AI Waifu community's work on personality serialization may lead to mind uploading, with human memory fitting in 1 TB. - [1:55:29] "I'm still within this terabyte range because frankly, that's all we need." ## Speakers - **Eugene** (host) ## Topics Language Models ## Mentioned Eleuther AI (company), Hugging Face (company), Stability (company), UILicious (company), A100 (product), Brain.js (product), Copilot (product), Falcon (product), GGML (product), GPT-Neo (product), GPU.js (product), Llama (product), LoRA (product), ONNX (product), PyTorch (product), RWKV (product), Raspberry Pi (product), Salesforce Code Gen (product), Selenium (product), Stable Diffusion (product) ## Transcript ### Intro **Host** [0:02] Okay. So I'm here with Eugene. Uh, we are in Singapore. We-- This is the first time I'm po-podcasting in Singapore. It's the first time I'm podcasting with my Singaporean accent. Uh, Eugene has been a very valued part of our latent space Discord for a while. **Eugene** [0:18] Mm-hmm. **Host** [0:18] And also diving deep into RWKV. I think you're actually the first person that brought it to my attention as, like, a potential transformers alternative. Um, you're also CTO of UILicious. **Eugene** [0:29] Mm-hmm. **Host** [0:29] Which is a UI testing company that's, uh, in Singapore here. **Eugene** [0:32] Yep. **Host** [0:32] Uh, anything else that you would flag out as, like, your high-level intro? **Eugene** [0:36] Um, like, what brought me into AI and machine learning is actually I started-- I originally wrote GPU.js, so that allows you to run JavaScript code on the GPU. This was pre-neural network boom. **Host** [0:49] Mm. **Eugene** [0:49] And my project got picked up by Brain.js and merged in, and that's how I actually went to the mad rush that is neural networks, and then now subsequently large language models. **Host** [1:00] So okay, let's talk about that a little bit. Um, what was the origin story for GPU.js? **Eugene** [1:05] Um, th- so the origin story for GPU.js is that, um, me and my friends, uh, at NUS, um, the, the local university here, we just wanted to run JavaS... I think it was, like, the era where everyone's just trying to do everything on Node.js and, and, and NPM packages, and we were just like, "Let's just-" **Host** [1:22] Was this, like, 2016, '17? **Eugene** [1:24] Yeah. It's quite far back. And then we were like, "Let's just do this for fun. Let's just prove that you can run JavaScript on a GPU, just because it should be faster, theoretically, for matrix multiplications." **Host** [1:35] Mm-hmm. **Eugene** [1:35] This, like, foreshadowing. **Host** [1:37] Mm-hmm. **Eugene** [1:38] And, and, and it was meant to be a joke that, yes, you can run JavaScript on anything. Uh, and we, and we managed to get it to run it for that very narrow case of matrix multiplication. It outperformed the base V8 engine by running it on the WebGL. And- **Host** [1:53] By a lot? **Eugene** [1:54] Bit of a-- Especially when you scale past 2,000 dimensions. Um, there is a gotcha because you have to transfer your, your variables from- **Host** [2:05] Yeah **Eugene** [2:05] ... from the JavaScript space- **Host** [2:06] Serialization. Yeah **Eugene** [2:07] ... to, to the GPU space. So anything less than 1,000 by 1,000, it tends to be not worth it. And then we just let the project just sit there on the internet. And it just sat there for one whole year until neural networks, it came, uh, it came in full steam, and someone picked it up and clubbed it together, and it's like, "Hey, we can train neural networks in the browser in JavaScript." And that is, that's how, how Brain.js grew on top of GPU.js. **Host** [2:35] Right. And just because I have a little bit of background to this, I actually still don't know what specific APIs. Are you using WebGL, uh, to, like, like... Are you, are basically abusing WebGL to a- get access to the GPU? Like, how do you get access to GPU, basically? **Eugene** [2:49] Oh, it's not really so much of an abuse. So the abuse, the crazier abuse part is actually upfront. So what, what we actually do is that when you submit a JavaScript code to, to GPU.js to execute in parallel, so y- uh, I, I think, I think you can just view it as a very common reduce function. **Host** [3:06] Mm-hmm. **Eugene** [3:06] So you have your function and then your data. So you've got your large data arrays, you put it in there. Uh, what happens is we serialize your function into, into code, and then we, we, we j- uh, do a ana- uh, analysis on it. So on it, and then we tr-translate that into WebGL code. So we had to implement a lot of things that were in JavaScript that were like, uh, shader code. At that point, it's still considered shader code, uh, that did not have support for. So for example, if you h- if you want to do a large, a large number manipulation and we only had small floats in the system, what we do, we just had two floats, and then we just abuse the heck out of it. **Host** [3:46] To simulate a big int or- **Eugene** [3:48] Yeah. Things like that. **Host** [3:49] Okay. **Eugene** [3:50] So, so that's in essence what the GPU.j-JS library did, is that we took your code, uh, accept, uh, abstract syntax tree, analyzed it. We figure out what it does. **Host** [4:00] Oh. **Eugene** [4:00] Then we rebuild the code in WebGL. **Host** [4:03] Okay. Why not, uh, have... So, so why, why this f- So this is a compiler? **Eugene** [4:08] Yeah. It's a... **Host** [4:10] Why the compilation approach instead of, like, a library approach where people can just kinda use functions- **Eugene** [4:15] We- **Host** [4:15] ... that you've made? **Eugene** [4:16] I think the m- it's back to the original goal of making it a joke more. **Host** [4:19] Right. **Eugene** [4:20] Just like- **Host** [4:20] Just to run JavaScript on the GPU **Eugene** [4:21] ... literally run JavaScript. **Host** [4:23] Okay. Okay. Yeah. **Eugene** [4:23] So we didn't want you to need to learn new commands and things like that. **Host** [4:28] Yeah. That's pretty crazy. Yeah. Yeah. Okay. So, and, and, uh, just to-- And because I had this initial confu- uh, confusion, Brain.js, uh, has nothing to do with TensorFlow even though I think both were run by Google or used by Google project? **Eugene** [4:41] No. Uh, Brain, Brain.js is not run by Google. It's more of a community-driven project. **Host** [4:46] Okay. **Eugene** [4:46] So, and I think it's commonly confused with TensorFlow because, like, let's be realistic. If you want to train real models, you're not gonna train it on JS. You're gonna train it directly with CUDA and what so on, because it just performs much better. But there is a benefit of running it purely in a browser because you ma- you make it completely possible for, like, teachers. Uh, and yeah, in fact, one of our most popular users were teachers teaching students on how to make neural networks. And the, the barrier of entry is no, it's not you need a CUDA, you need a setup. No, you just need your browser. **Host** [5:21] Yeah. **Eugene** [5:21] Which makes it significantly easier, even though it's all toy models. And in that use case, TensorFlow.js and Brain.js is functionally the same with just different APIs, at least for serving this target market. **Host** [5:32] Yep. Yeah, yeah. It, the, I mean, it's s- the best user experience for sandboxing. **Eugene** [5:37] Yeah. **Host** [5:37] Um, you're just spinning something up without dependencies. Um, okay. And then so fast-forward, uh, after, uh, GPU.js, uh, what else did you get up to? **Eugene** [5:46] So after GPU.js, that's where I move on to running my own startup, so UILicious. I guess m- that was because I was, at the time, professionally working for banks and pr- and private institutes. Uh, and I'm surprisingly like, a l- um, for me, it's like why we have so much high tech applications, but at the end of the day, we are just testing a lot of things manually, and I just wanted to automate that, and that is why I started a, effectively a test automation company. And, and even then early on, we actually tried to automate things more, uh, with AI even. But we found that at least at that time, it was not ready. And fast-forward, so we built a product around it where you can automate your browser using low-code. Just go there, type simple commands, go to Google, click on this text, run. Um- **Host** [6:32] Which is another compiler, li- compiled language, right? **Eugene** [6:35] Yeah. **Host** [6:35] You had your own, uh- **Eugene** [6:37] Oh, this actually in JavaScript **Host** [6:38] ... testing language. Oh, really? **Eugene** [6:39] It's, it's a JavaScript library, but we focus on making it easy for manual testers. **Host** [6:45] Right. **Eugene** [6:45] So we-- So like, if you see all the existing, let's say, browser automation libraries, they, they are all heavily async based. **Host** [6:51] Yeah. **Eugene** [6:52] Uh, teaching someone with zero programming skill how to deal with asyncs is a complete nightmare. **Host** [6:58] Yeah, yeah. **Eugene** [6:59] Um- **Host** [7:00] Yeah **Eugene** [7:00] ... so, so, so we make steps there. For example, we make it synchronous. We, we- **Host** [7:04] Yeah **Eugene** [7:04] ... we, we, we, we don't expect you to know CSS selector. We just ask you for your text on screen. **Host** [7:09] Yeah, yeah. **Eugene** [7:10] Then- **Host** [7:10] But it's, it's still JavaScript. **Eugene** [7:12] Yeah. Then that runs on, uh- **Host** [7:13] Yeah **Eugene** [7:14] ... Selenium, and then it does all that. So it's not AI, but the big jump for us was that subsequently, more recently, because we've been building our dataset, we s- we started having our own self AI on our platform where you can just describe your test and it will generate for you. **Host** [7:29] Right. **Eugene** [7:30] Including hallucinations. **Host** [7:33] All right. So lots of fun. Um, yeah, and so, so how did you-- So you, you were running UILicious, which is low-code platform. Uh, I got the first demo maybe four years ago. **Eugene** [7:42] Yes. **Host** [7:42] Uh, and uh, I was like, okay, fine, you know, you're doing testing. There wasn't an obvious AI angle. I mean, now that you explained it, it was great. But like, what was your personal like, okay, I'm gonna be the dedicated AI guy for your UILicious? ### Alternatives **Eugene** [7:55] Oh. Mm, I think because for most part we knew that... Okay, so one of the things int- one things that I found very interesting with the huge transformer boom right now, right, is that, is that traditionally, right, traditionally, right, and I think I have an article on this also, is that, is that like when you tell companies that you need to-- when you want to build your own AI, you need a really large dataset. **Host** [8:18] Mm-hmm. **Eugene** [8:18] And, and over time, actually, the amount of datasets that you need is actually scaled down because you can just now fine-tune- **Host** [8:26] Foundation models. **Eugene** [8:27] Yeah. Fine-tune on foundation models. And, and when we started UILicious, we always knew at that time because a lot of our com- other companies that were launched at the same time were dealing with neural networks, that at some point the data that, uh, we've been collecting data on, let's say, how to do testing website, it's just a very specific focus. Um, um, basically every single test that has run on our platform, unless our cl- customer has opt out or delete their account, uh, basically privacy related stuff, we actually still retain the test data. **Host** [8:58] Mm-hmm. **Eugene** [8:58] And that's something that, uh, uh, that we always felt that was useful in the long run to be able to actually build a huge training model. The irony of that was that even though we were building all those datasets, as, as the threshold came in and tran- and the transformer boom happened, we realized we don't actually need that big of a dataset anymore to actually get- **Host** [9:16] Right **Eugene** [9:16] ... a functional AI to do- **Host** [9:17] Can, can you give order of magnitude? Like what were you expecting, and then what did you find? Like are we-- How, how off are we? Uh, you know, like do you need like millions of, um, I don't know, customer of, of test data or-- and then, and then you found that it was just thousands or, you know, like just- **Eugene** [9:34] It's literally- **Host** [9:35] ... quantify, quantify something like that. **Eugene** [9:36] So, so, um, so and I think this is actually one of the, one of the key insights for actually, especially for people who is like trying to build on top of transformer models- **Host** [9:45] Yeah **Eugene** [9:45] ... or their commercial. Um, pre-transformer, uh, large language models is, is we would-- we would always be thinking of like in terms of like 100 gigabytes of data, one terabyte of data. **Host** [9:55] Right, right. **Eugene** [9:55] Like millions, multi-million dollar, millions of record for all the different examples. Post transformer, um, it's literally you probably need only like 1,000 or 10,000 enough like data that you can literally get an intern a few weeks- **Host** [10:11] Right **Eugene** [10:11] ... to just get it done. And you have a working model. It may not be that great, but frankly, every piece of data you add after that is a, is a diminishing, uh- **Host** [10:21] Sure **Eugene** [10:22] ... returns. **Host** [10:23] And, and it's, it's specifically structured as, I mean, 'cause it's, it's a language model. It doesn't actually have any inherent understanding that it is automating the browser. **Eugene** [10:31] Yeah. So- **Host** [10:32] So it's presented as like a prompt answer pair, like question answer pair. **Eugene** [10:35] Correct. **Host** [10:35] Yeah. **Eugene** [10:36] So, so typically, so for at least for our internal model that our users are using is, it's presented as, here's the prompt, describe your test or what you want to modify the code. It-- and then subsequently generate the code for you. **Host** [10:47] Right. **Eugene** [10:47] So it's, it's, and now in hind- in hindsight, it's now basically Copilot. **Host** [10:52] Yeah, yeah. **Eugene** [10:53] Yeah. I, I think now Copilot is adding that chat widget. Are they fully launched yet? **Host** [10:57] Yes. **Eugene** [10:58] Yeah. **Host** [10:58] Uh, I actually downloaded it yesterday. I haven't, um, actually used it yet, but, uh, it, it is a separate VS Code extension. So there are now, there are now three Copilot extensions shipped at GitHub because they have shipped their own chart. Uh, I'm, I'm friendly, I'm very quite friendly with that team, but uh, it's very funny. Um, but just to come back to you. So did you implement this with GPT-3? Is that- **Eugene** [11:19] So- **Host** [11:19] Is that where it was? **Eugene** [11:21] Um, so what we implemented, uh, what we trained for, at least our code model, we based it off the Salef- Salesforce Code Gen model. **Host** [11:28] Okay. Right. **Eugene** [11:28] So that was our, the foundation model that we built on top. We are looking into replacing it in parts, but that becomes a longer conversation. **Host** [11:36] Code Gen being the first really credible open source code specific language model- **Eugene** [11:44] Correct **Host** [11:44] ... that was released by literally anyone, I think about three years ago. **Eugene** [11:47] Yeah. **Host** [11:48] Uh, and then they recent- recently released Code Gen 2. **Eugene** [11:50] Correct. **Host** [11:51] Uh, any opinions on Code Gen 2? And just while we're, while we're on this topic. **Eugene** [11:55] I actually think... No, so in terms of like code gen, one big appeal for the code gen and even code gen 2 model is that, uh, Salesforce took a very clear and clean approach to the licensing. **Host** [12:07] Meaning, uh, they were very, very clear that everything that they trained on was, uh, open source. **Eugene** [12:12] Yeah. MIT. **Host** [12:13] Yeah. All these open source. **Eugene** [12:13] They did, they did, they didn't touch the, the problematic licenses. **Host** [12:16] Yeah, yeah, yeah. **Eugene** [12:17] So-- And I-- You can imagine it was- **Host** [12:19] And you, do you think that Copilot did? **Eugene** [12:21] Uh. I'm-- Knowing Microsoft's statement on how liberal they were about GitHub data- **Host** [12:28] Uh-huh **Eugene** [12:28] ... and they were saying-- they used the term that it's under fair use. **Host** [12:33] I see. **Eugene** [12:34] Uh, yeah. **Host** [12:35] I see. Okay. **Eugene** [12:35] I, I have no reason to believe that they didn't. **Host** [12:38] Right. **Eugene** [12:38] But, um, this same problem happens to actually a lot of existing, uh, code gen models. And, and that was actually the main appeal for me, uh, for running, for actually building on top of the Salesforce code gen model. Uh, mostly also because like, like for us, we deploy on-premise into enterprises- **Host** [12:56] Yeah **Eugene** [12:56] ... in Europe, and they ask questions. **Host** [12:59] Right. So what, what, what does this deploy on-premise mean? Like you, you, you pack UILicious into a container and you- **Eugene** [13:05] Yeah **Host** [13:05] ... you give it to them? **Eugene** [13:06] Yeah. And they see this. **Host** [13:07] And there's a, there's like a license fee or something. **Eugene** [13:08] Correct. **Host** [13:09] Okay. Cool. Um, that's very interesting. Yeah. Uh, okay. I, I don't know if I have any other questions on-- based on that. Uh, yeah. Anything else before we go into alternate, like the reasons for- **Eugene** [13:20] Yeah **Host** [13:20] ... alternative models? **Eugene** [13:22] Mm. Okay. So, so I-- Anything else do I have there? No, I don't really have much else. Uh, for alternative models, so yeah. **Host** [13:29] So let- maybe let's set the premise, right? Like, um, transformers have won- ... for now. **Eugene** [13:35] They have slipped the neural networks. **Host** [13:37] Uh, yes. Um, and, uh, you know, it, it, it seems like you have had, had a history since with machine learning since before transformers, and now they're, they're at kind of like the peak of their power. Um, uh, and I, I see that, you know, there's, there's a desire for alternative models for a number of reasons. Um, but I'm try-- I'm very curious as to what drives your personal interest in alternative models. **Eugene** [14:00] Uh, so, um, so first things first, to be clear, majority of our AI is still based on Transformer, at least within my company. **Host** [14:06] Yes. **Eugene** [14:07] Uh, but what drove me into alternatives beyond Transformer, in essence, uh, once we actually managed to get our bot to generate UI testing code, the most obvious next thing that our customers started asking, "Hey, um, this-- Let's say the test failed. Can you ana- uh, can your bot now, uh, can your AI now analyze my website and then tell me what's wrong and tell me what to change?" **Host** [14:31] Okay. **Eugene** [14:31] Basically, basically they're getting lazier and lazier in, in their- **Host** [14:33] Yeah, yeah, yeah. **Eugene** [14:34] And, and that's to be expected. **Host** [14:35] Humans are very good at moving goalposts. **Eugene** [14:37] Yeah. And, and we-- And I was like, "Okay, uh, yeah, that-that's something I'm w- I was working on." And, and, and we had something working for toy websites. Uh, but the first thing that, the first thing that we hit was that we started, uh-- One thing that we do, and internally, is that we look at the, I think what was the list? Top 100, top 1,000 websites. **Host** [14:59] Okay. **Eugene** [14:59] And we basically just run, or we actually do run our, our test platform against it to see, make sure that our code works against like any front-end platform. **Host** [15:07] Well, what do you mean run your test platform, right? Because you don't have tests for them. **Eugene** [15:12] Uh, yeah, we have some very rudimentary basic tests. **Host** [15:14] Yeah. **Eugene** [15:14] Like go to website, see something, click something, add to cart. **Host** [15:17] Okay. **Eugene** [15:17] Yeah. That's it. **Host** [15:18] That's it. **Eugene** [15:18] Uh, the, the idea is more of like, because there are so many frameworks out there, uh, and our- **Host** [15:23] You just wanna make sure you cover all of them. **Eugene** [15:24] Yeah. And so we did the same thing for our AI, and the first thing that it, it died on was literally Amazon. **Host** [15:31] Why? Oh, five megabytes. **Eugene** [15:33] Yeah. I, yeah, I think you heard me mention it. Uh, so, so it, um-- When you are trying to analyze a website, it's like e- uh, even, like we are talking about-- we've been talking about increasing token count size, right? But for e-commerce websites in particular, even if you strip off the CSS, even if you strip off the JavaScript, having the entire HTML in megabyte size is not unheard of. **Host** [15:54] Yeah. **Eugene** [15:55] And that's where it's like, how am I supposed to solve this in terms from an AI point of view? **Host** [16:01] Okay. W-well, how many tokens would that be? Like... **Eugene** [16:03] Oh my gosh. Easily- **Host** [16:05] I mean, today-- for today it's nothing, right? Like 10,000 tokens, you know? **Eugene** [16:09] No. **Host** [16:09] It's not that much, right? **Eugene** [16:10] No. Because, okay, the tokenizer doesn't do very well with HTML content. **Host** [16:15] Oh, right. Okay. **Eugene** [16:16] So you could easily be looking at over a million tokens. **Host** [16:19] I see. **Eugene** [16:20] But- **Host** [16:20] Which is still too much even for today. **Eugene** [16:22] Yeah. **Host** [16:22] Um, did you look into making your own tokenizer? **Eugene** [16:26] Um, that's some- that's something that we explored. Uh, we mo- I think what we found more realistic was to actually parse the HTML into a more token-friendly format. **Host** [16:35] Yeah. Right. **Eugene** [16:36] So this way, this way we can still build on top of existing models. Uh, but yeah, we, we are exploring that as well. But, but back to the alternative route. So, like the key things for me and, and was at that point, and as-- and subsequently, I think I showed you like the experiments with like English compiler and things like that, right? Uh, AI agents generating code. You also have your own, uh, small there. **Host** [17:01] Mm-hmm. **Eugene** [17:01] Was that, was that the context size is a real problem, and transformer inherently by its nature, at least vanilla transformer, I know there's transformer XL and some other attempts, uh, is that it quadrat- quadratically scales with the context size. **Host** [17:19] Mm-hmm. **Eugene** [17:19] So we-- So if we scale to like, let's say 100,000, that's already requiring a shit ton of compute and VRAM, and I don't even want to imagine what happens to 1 million or 10 million. And, and that's, that's where I, I needed-- I was like, "Okay, this is a fundamental problem that needs to be changed. Uh, if not, we will not go past this." And I think there's also now like a lot of people who are very interested in like models that can handle large context size because they also want it to be able to use in use cases where they never, never need to fine-tune, because fine-tuning is a pain apparently. **Host** [17:54] Yes. Uh, that said- Okay. Well, there, there's, there's, there's issues with just throwing everything in context, right? **Eugene** [18:00] Yeah. **Host** [18:00] Like the, it's shown that, um, retrieval is only best when the, the item that's relevant is in, in front or at, in the, at the back of the context window. Um, so it's ... Basically, I'm just like, maybe we've just tapped out. Context is working memory, and maybe it's like turn-- maybe transformers are very similar to humans in that our working memory is only of a given size. If you try to artificially extend it, you would just have ... You would just make it very lossy. **Eugene** [18:23] Yeah. So, so, so that's where I end up landing on the RWKV model because in that sense, right, so, so you ... One thing that I always found very weird for transformers, but the, I mean, it's by design, is as you infer each token, you are, you are recomputing everything up, right? **Host** [18:42] Mm-hmm. **Eugene** [18:42] That's the quadratic part. And, and, and while, while you, while you're mentioning about the working memory problem, uh, in theory, with enough attention hits on this **Host** [18:53] Mm-hmm **Eugene** [18:54] ... and, and, and people seems to be trying to cram more and more attention hits into the process, it could scale that way, ignoring compute cost. **Host** [19:01] Okay. **Eugene** [19:02] And yeah, ignoring compute cost is like, just like a very liberal, "Let's just throw as much H100s." It doesn't make sense. But, but the, uh, RWKV what is still, was fundamentally a neural network at its core. It ends up scaling linearly as it goes through the tokens. It also s- it will still suffer from the memory issue. So, so in, so like, within the RWKV, we do m- we do, like, measure two separate things. So like one, we call it the perfect memory. So, uh, in, um, the model will have only a certain amount capacity where it can remember things perfectly, just like humans. And then there will be then beyond that, that is where it will start to discard things from its perfect memory. **Host** [19:44] Right. **Eugene** [19:45] And, and I felt that this was actually a lot more in line with our goals and commercially and also, um, what I felt was that, uh, was more useful in the long run because it's cheaper compute and it could be potentially paralyzable over a long run. **Host** [20:01] Right. Um, so we're going to go into r- RWKV paper in a, in a bit. **Eugene** [20:05] Mm-hmm. **Host** [20:05] But one thing I wanted to ask, um, you kind of glossed over how you found it in the first place. ### Discovery **Eugene** [20:09] Oh, how did I- **Host** [20:09] Because you're not a researcher. You're not like ... It's, I don't, I don't imagine you're, like, reading papers every day or something. Um- **Eugene** [20:16] Until recently. **Host** [20:17] Until recently. Um, how do you find it? **Eugene** [20:19] How did I find it? Uh- **Host** [20:21] And how do you know, like, this is the one to bet on versus, uh, there's a bunch of other alternatives, right? **Eugene** [20:25] I think what was quick, I think it was rather quick after I concluded that transformer as it is will not scale to 10 million tokens. **Host** [20:39] Okay. **Eugene** [20:39] And- **Host** [20:40] And, and so by the way, you, you mentioned Transformers XL. **Eugene** [20:42] Yeah. **Host** [20:42] Uh, we also did a, a episode on flash attention- **Eugene** [20:45] Gotcha **Host** [20:45] ... which, which helps to, uh, make part of it sub-linear at least. **Eugene** [20:48] Yeah, but that is like way, way after I already dived into- **Host** [20:52] Right **Eugene** [20:52] ... RWKV. So history-wise, at that point in time, transformer ex- we are talking about like the, the when 4K was the limit everyone knew. **Host** [20:59] Right. And this was last year. I mean- **Eugene** [21:01] Yeah **Host** [21:01] ... just to set context. Okay. **Eugene** [21:04] Yeah. **Host** [21:04] Um, okay. And then, um, yeah, so, so a- and you just kind of were searching around, you found R- R- other, other RWKV. Uh, presumably, like, did you, do you go straight into the Discord? D- was it like primarily a GitHub repo? Like, what was it? Because as far as I can tell, there was no paper until maybe about two months ago. **Eugene** [21:24] Oh, and I pre- and I talked about it before the paper, right? **Host** [21:27] Yes. So you found it before they did any publicity. **Eugene** [21:30] Mm. **Host** [21:31] Which is weird. That's not normal. **Eugene** [21:34] Fair enough. **Host** [21:34] So what, what happened? What did you do? **Eugene** [21:36] So, uh, what, what I did ... Okay, so it was basically, uh, I, I believe ... Okay, so it, it's a mixture of things because it's like be- I was searching be- GitHub, uh, searching like, uh, forums, um, other Discords, and also like blogs actually. **Host** [21:52] Can you shout out which, which Discords and which forums were super helpful to you? **Eugene** [21:55] Uh, super helpful will be mostly in Lu- in Lucas forum, Discord itself. Blogs, blogs, it's very hard to pinpoint to it because at that point in time it was just like- **Host** [22:05] Random people's blogs. **Eugene** [22:06] Yeah, just ... I was just like getting all the- **Host** [22:08] Yeah **Eugene** [22:08] ... because everyone was just creating lists of lists, right? **Host** [22:11] Yeah, yeah. **Eugene** [22:11] I'm just, and I believe you also have a list of this somewhere. **Host** [22:14] Yeah, but mine is very ... So I would con- consider myself very trad- **Eugene** [22:18] Mm-hmm **Host** [22:18] ... in the sense that I, I would just follow the large model labs. Whereas the kind of list that you have to follow in order to get to something like RWKV before they'd done any publicity is the non-trad, like, uh, you know, the kind of people that is not working on- **Eugene** [22:32] Ah, okay **Host** [22:32] ... NewsRMAs, Wizard, you know, that like no credentials. I don't even know who the hell they are, but they're just working on it. **Eugene** [22:39] Oh. So, so, so, so, so this, uh, the, the list, what ... Okay, this is all foggy memory, and I might be hallucinating this because y- there was too many lists. But I believe the list that actually what brought me to RWKV was, was that beyond ... So this is something, this is, this is a topic that we can actually touch upon later, right? Uh, beyond OpenAI's model and, and, uh, beyond ChatGPT and Claude, the, the two big models, right? Outside of the English-speaking nations, right, a lot of the open source models really fall flat. **Host** [23:14] Mm-hmm. **Eugene** [23:14] And that is, and, and that is why like, uh, when, when, when you, when you actually go through like lists for, uh, for like doing things with in other languages- **Host** [23:24] Mm-hmm **Eugene** [23:25] ... RWKV actually stood out and in point. **Host** [23:27] Mm. **Eugene** [23:28] And, um, just on the basic premise, and, I mean, we're not even talking about architectural advantage. It's just the basic premise that they imported the dataset in other languages in the training data. **Host** [23:38] Yeah. **Eugene** [23:39] And- **Host** [23:40] Was that a dr- 'cause I mean, I pr- I imagine 99% of your customers are English. **Eugene** [23:44] Yeah. **Host** [23:44] Was that really a driver for you? **Eugene** [23:46] It was a driver, but this- **Host** [23:46] Or you're just trying to explain it. **Eugene** [23:47] Yeah, that's how I landed onto like all these blogs and article. **Host** [23:51] And, and can you say if, when you say fall flat, the main one that I know about is, uh, there's a tokenizer penalty for non-English. **Eugene** [23:58] Yep, that's it. **Host** [23:59] Right. So like Ch- uh, Chinese is up to s- Chinese or Japanese or Thai or something like, it's like 16 times the number of tokens For, for a, for a typical English sentence. **Eugene** [24:08] Yeah, but even before that, right? Because, I mean, yeah, I think you understand, like co- a lot of community users, they want to not use the commercial APIs. **Host** [24:16] Okay. **Eugene** [24:17] So they try to find open source models. **Host** [24:18] Yes, and we'll talk about the not safe for work, uh, people. We, I really want ... 'Cause you- you've actually talked to them. I have never talked to these people. **Eugene** [24:25] Okay. **Host** [24:25] But, like, when I sh- discovered them, they have ... It's a huge community, they're extremely passionate, and they're actually good. **Eugene** [24:31] Really. Yeah, they're really good. **Host** [24:32] They're good. They're good at this. So let's talk about them, right? **Eugene** [24:35] Yeah. **Host** [24:35] But yeah, we can talk about that later. **Eugene** [24:36] Yeah. So, uh, so, so they don't want to use the commercial models, um, and they want to use the open source model. And it ... There is a tokenizer penalty, which is true, but I think on the more fundamental basis, right, if you look through the datasets, and, uh, and this is also partially in fault because the way we set up our evals, all our evals are written in English. **Host** [24:59] Mm-hmm. **Eugene** [25:00] And at least for the majority of them. And if we are racing towards building AI models, at least right now, yes, you see all the companies as they build their open source model, and they just want to narrowly focus on the evals. Adding in a foreign dataset is actually a loss. Because as ... Once you are below a certain param count, so we're talking about- **Host** [25:19] Listen **Eugene** [25:19] ... like seven and 14, right? **Host** [25:20] Yeah. **Eugene** [25:22] The more you add that's not in line with your evals, the more it will degrade. **Host** [25:26] Yep, yep. **Eugene** [25:27] And, and they just exclude it. **Host** [25:29] Yeah. **Eugene** [25:29] And so the model just- **Host** [25:30] The, the priority is English. Yeah. **Eugene** [25:31] Yeah. **Host** [25:31] I get it. **Eugene** [25:31] The, the model just fundamentally didn't support. **Host** [25:33] Wait, so what's the trade-off? Like, I mean, okay, so English and Chinese or ... You know, there's- there's all these other languages. Uh, what do you pick, like? **Eugene** [25:41] So, uh, so RWKV started w- uh, started with, uh ... Al- also for, for, in context, the main person leading the RWKV project, Blink, he's from China. So he, he naturally has an interest to make sure it supports Chinese. **Host** [25:55] Of course. **Eugene** [25:56] Yeah. So English- **Host** [25:57] And, and there are a fair amount of, uh, bilingual models especially- **Eugene** [26:00] Correct **Host** [26:00] ... that are English and Chinese from the major universities in China. **Eugene** [26:03] So, so we started from, uh, basically English, Chinese, Japanese, Korean. Um, frankly, this is large part mostly because there were fans in those communities- **Host** [26:13] Uh-huh **Eugene** [26:13] ... that came on board. And then, and then subsequently we tried to onboard other languages as well. Uh. **Host** [26:19] Yeah. And, but these people are, like, again, they're not researchers. **Eugene** [26:23] Nope. **Host** [26:23] Uh, no money. **Eugene** [26:25] Nope. **Host** [26:25] Like, training on their, their home GPU lab or whatever, right? **Eugene** [26:28] Uh, partially true, but, uh, so, so, so how, how this works out, right, so for the RWKV model. At least how I see it works out for a lot of the, uh, other languages was that we have the foundation model. Uh, and this is the fu- this is the foundation model where we just kind of say, "Evals be damned." "Let's just make sure to include all the other languages." **Host** [26:48] Okay. **Eugene** [26:48] And, and, and when we included the other languages, right, uh, the model works for most parts- **Host** [26:56] Mm-hmm **Eugene** [26:56] ... uh, for the other language. Subsequently, these individuals who wanted to use their, these models in, for their respective use cases, we will then fine-tune respectively. **Host** [27:06] Hmm. **Eugene** [27:07] Because it's easier to fine-tune in another language for your use case than ... I mean, this is classic fine-tuning than- **Host** [27:13] Yeah, yeah **Eugene** [27:13] ... than to train that language from scratch. **Host** [27:16] Okay. **Eugene** [27:17] And I think m- more recently, and this model is not 100% trained yet. But more recently, uh, we, RWKV has released the, what we call the world model, where, where we go the next step, uh, of even including all the translation datasets that we can find, even for minority languages that people, and in our Discord. 'Cause the goal for them, the goal, the long-term goal for us at least internally is so that we wanted an AI model for everyone. And everyone does not mean USA. It means the world. **Host** [27:49] Yeah. Well. **Eugene** [27:50] So there are a lot of languages in there. **Host** [27:51] Well, is it a, is it Asia biased or ... You know, I, uh, give me a sense of like- **Eugene** [27:57] It's probably, no offense, it's probably still gonna be US biased in terms of, like, knowledge. **Host** [28:02] Okay. **Eugene** [28:02] Because what we are doing is still power of red ge- red pajamas for the knowledge. **Host** [28:06] Okay. **Eugene** [28:07] But in terms of language, we add all the other languages, Wiki and translation set. So it's ha- ... I mean, we haven't ev- fully evaluated the bias yet, but I'm quite sure that when disproportionately knowledge is still within the English universe, there's the, the bias is there. But frankly, we are still at the stage where we can support the other languages. **Host** [28:27] Yeah. **Eugene** [28:28] And, and, and, and I think I mentioned this. This, this, this, one of the interesting parallels that, that sometimes I have, right, is that I can be in the-- I can see in the Eleuther forums and all that, and then we are talking about alignment and, and, like, we're talking about it in very big- **Host** [28:41] Which is, yeah, very keen on safety and all that, which is great, but, like, it's not your goal as- **Eugene** [28:47] Yeah **Host** [28:47] ... as the RWKV community. **Eugene** [28:48] Yeah. And, and when you, when you talk to, like, members of the community that came on board, they said, "Oh, I want to get this to work for Korean, Japanese, Thai, uh, Arabic languages," and, and so on, so forth. They just want something that worked. **Host** [29:02] Yes. **Eugene** [29:03] They don't want it to be ... They, they are not after the big model that does everything. They just want something that they can play with in their language, and that was very important to them. **Host** [29:12] Yeah. And these are, uh, literally just hackers, uh, doing it for personal enjoyment. **Eugene** [29:19] Correct. **Host** [29:19] Uh, n- not yet for work. **Eugene** [29:21] Yeah. **Host** [29:22] Or maybe some of them for work. You, you don't know. **Eugene** [29:24] We don't know. **Host** [29:24] Yeah. Okay. **Eugene** [29:25] I mean, the core character AI category, there's quite a number of them using it for, for that, so, so professionally. **Host** [29:35] Professionally, okay. Uh, as in they, they run character companies- **Eugene** [29:38] Yeah **Host** [29:38] ... let's, let's call it. Okay, cool. Yeah, yeah. I, so I, uh, you know, I'll, I'll signal that I'm interested in doing, like, a AI Waifu episode. Um, and I need to find, like, the perfect, like, someone, someone doing that and to, to just explain everything that they found. Actually, I, I'm very interested in, like, basically pairing this with a psychology professor who can ask, like, psychological questions about, like, what have, what have you found about human sexuality and human behavior when they're just talking to a AI bot. Um, I think it's very ... I don't know. I think no one's covering this. So I listened to, like, I actually listened to a few psychology podcasts and, um, they have, they, they are completely out of the loop. They, they're not even aware that this is going on. And it's so huge. **Eugene** [30:17] Right. **Host** [30:17] It's like literally millions of people, right? **Eugene** [30:19] Yeah. So you're not aware about people using AI, I guess, in the form of therapy? **Host** [30:25] Yeah. **Eugene** [30:26] Or personal companionship? **Host** [30:27] Well, um, they're not talking about it. **Eugene** [30:29] Oh. Okay. **Host** [30:32] It's maybe not polite conversation, especially 'cause it's, like, not safe for work. But, like, I think it's just an emerging category that is interesting. **Eugene** [30:39] Yeah. Um, especially, I mean, just gonna be cut straight to the chase, especially Japan. **Host** [30:45] Yeah, yeah. Yeah. Um, well, and then there's also, like, um, we always say AI Waifu. But actually, uh, always call this, like, AI Husbando. Uh, it's actually- **Eugene** [30:54] Yeah, there's that too **Host** [30:55] ... it's bigger. **Eugene** [30:57] Bigger? **Host** [30:57] Yeah. **Eugene** [30:57] Oh, I didn't... I was not aware about market sizes. **Host** [30:59] It's bigger. Yes. I've actually looked into this. **Eugene** [31:01] Oh. **Host** [31:02] Uh, and so I can resolve this, uh, with a very, very simple example that everybody will understand, right? Um, Amazon Kindle Unlimited is the subscription service, uh, where you can just, you know, pay a monthly fee and get all the books you want. **Eugene** [31:15] Mm-hmm. **Host** [31:16] What sells the most? **Eugene** [31:18] Com- romance models. I mean, romance novels. **Host** [31:21] For women. **Eugene** [31:22] Oh. **Host** [31:22] 'Cause they like to read- **Eugene** [31:24] Okay **Host** [31:24] ... about romance. **Eugene** [31:26] Oh, that makes a lot of sense. **Host** [31:27] Men are visual, women are verbal. **Eugene** [31:30] And in this case- ... language models are text. **Host** [31:33] Yes, exactly. **Eugene** [31:34] I mean, we do try to... They, uh, they do try to dress it up. **Host** [31:37] Yes, yes, yes. Okay, cool. Um, so I think it's, it's, uh, it's great. Uh, let's... Shall we pause here, and then I'll switch to s- the screen? **Eugene** [31:43] Sure, sure. **Host** [31:44] Okay. Um, ### RWKV Overview **Host** [31:47] all right. So we have it, uh, pulled up. Uh, we are gonna screen share for the bulk of this, so if you're listening on audio, might be a good time to switch to the YouTube channel. Um, so we're just gonna start with an intro. What is RWKV? **Eugene** [31:59] So RWKV is a modern, uh, recursive neural network with transformer-like level of LL- LLM performance, which can be trained in a transformer mode. And this part has already been benchmarked against GPT-NeoX in the paper, and it, it has similar training, uh, uh, performance compared to transformer models of the same dataset and parent count, so specifically the GPT-NeoX model. So, um, the, the key thing is that even though it's matching in performance while training blo- training both the GPT-NeoX, it's doing all this without attention layers. And in the process, right, it's actually having a much substantially lower compute based on its design, and also because it's a neural network, which we'll dive into later why, uh, why that's substantially lower, uh, in both training and inference. And this back to, like I mentioned previously, transformer, at least traditionally transformer until, until we found out about Transformer-XL and things like that, tends to scale quadratically based on the context size. And this applies not just in inference but in training. **Host** [33:05] Mm-hmm. **Eugene** [33:05] And due to how to, like, um, due to how this is still a neural network in its heart, even though it can train like a transformer, um, it's able to do so much more efficiently and faster, especially, like, when you hit context like of 8K, 16Ks and above. And, and once you do, like, quadratic and linear, the differences starts to, like, go- **Host** [33:25] Of course **Eugene** [33:26] ... crazy once, once you scale the numbers up. And that was the main benefits of the RWKV model, per se. There were a few prominent, uh, researchers when they actually reviewed through the RWKV paper when it came out. Um, they did highlight an important question of, of like, it... Is this, like, evidence to literally maybe all that really matters is that you need a large dataset and a scalable model. **Host** [33:51] That makes sense, obviously, to some approximation. Um, but you are still using attention? **Eugene** [34:00] No. We are, we don't use atte- attention inside. **Host** [34:02] Okay. Yeah. Let... Could, could, uh... Maybe, maybe let's rewind a little bit. Uh, so are- **Eugene** [34:07] Of specific attention as you understood it. Yeah. **Host** [34:10] Okay. Tell me, tell me, tell us more. **Eugene** [34:12] So, um, so we, we, we use weighted receptors and, uh, and- **Host** [34:18] And if there's any diagrams I should pull up, let me know. **Eugene** [34:20] Oh, okay. Uh, let's... Okay, so we are using AFT. Um, th- so this Attention Free Transformer, and this is, this paper was written by Apple. **Host** [34:30] What the hell is an Attention Free Transformer? Okay. This is un- this is unusual. **Eugene** [34:35] Yeah. So, so, so we, uh, we basically, we, we, we use the, the, the weighted retention weights, and we, it's, and we compute over it. And, and in essence, right, is this is like the classic, like, stacking more layers. Once you do on top of it, like, you don't really need- ... attention we- once you have enough weight and layers stacked on it. **Host** [35:05] Okay. **Eugene** [35:06] Uh, it, it, it, it... I don't know whether we want to go into the deep dive of AFT. **Host** [35:09] Sure. **Eugene** [35:10] But, but the, uh- **Host** [35:10] That's interesting. I've never heard of this paper. **Eugene** [35:12] Yeah. So, so this is, this is, this was written by Apple, and subsequently we integrate, uh, uh, at least Blink, uh, the, the creator of RWKV, took this, took this and applied it to a language model- **Host** [35:23] Mm-hmm **Eugene** [35:23] ... and scaled it up. **Host** [35:25] Right. **Eugene** [35:26] And, and that is h- and that is how, how we, we landed on RWKV w- that doesn't use attention. So w- uh, sometimes within the community we use the word light attention, because what happens is that these layers and d- and these weights will, will still play the role of attention. They will- **Host** [35:43] I was gonna say, you, you end up approximes- approximating attention. **Eugene** [35:46] Exactly. So, so it, it, it ends up, it ends up, like, looking at the tokens or parts of the memory and then applying it to the output. **Host** [35:54] Mm. **Eugene** [35:54] So well... And the key benefits is that, 'cause remember the attention models, the multi-hit part. **Host** [35:59] Mm-hmm. **Eugene** [36:00] It will need to scan all the tokens back and forth. **Host** [36:02] Yeah. **Eugene** [36:02] This removes that requirement. **Host** [36:04] Mm-hmm. **Eugene** [36:05] And hence it reduced the overall compute count. **Host** [36:07] Mm-hmm. **Eugene** [36:07] I might be jumping back and forth a bit, but that's the, that's the one of the key essence of the WKV segments. Uh, and we call it light attention. I... And this is the part where I would disagree with the RWKV community in s- in some parts. I think that was a bad name. **Host** [36:22] Ah, whatever. **Eugene** [36:23] Because, because it's cute. **Host** [36:24] Why is it a bad name? **Eugene** [36:25] Mm, um, this is the part where- Because when, when, when the RWKV paper came out, uh, RWKV paper came out, right, and then we talk about, like, we use this and we call it like attention, but by design, it's really nothing like your existing attention head models. And it end up, like, sidetracking the Hecker-Noon debate on, like, one corner. It's like, "No, this is technically attention, approximating attention." **Host** [36:54] Mm-hmm. **Eugene** [36:54] Then another group was like, "No, this is not attention." **Host** [36:57] I see. **Eugene** [36:58] And, and but I'm like, "Propose a better name," because I have no idea what to call it. **Host** [37:03] Okay. Um, what else should people know? Uh, maybe we can explain what it, what RWKV stand for. **Eugene** [37:10] Oh, uh, you have to open that in the paper. **Host** [37:14] Okay. **Eugene** [37:14] Um, recursive w- **Host** [37:15] I think the paper is in here. **Eugene** [37:17] Yeah. So reinventing RWKV. Yeah, receptive with the key values. **Host** [37:23] Okay. **Eugene** [37:24] Yeah. **Host** [37:24] And each of these are, like, actual things that you model in the code, right? **Eugene** [37:27] Correct. So, um, so we, we can go into that. Um, so- **Host** [37:32] Which attention historically is a query key value. **Eugene** [37:34] Correct. **Host** [37:34] Okay. **Eugene** [37:35] So, so, so do you want to jump straight into the layer architecture? **Host** [37:39] Uh, should we, should we cover something else first? **Eugene** [37:41] Mm. Is there anything- **Host** [37:44] I mean, we can... Any- anything, like- **Eugene** [37:46] High level, right? **Host** [37:46] High level. Okay. There's a 7B, there's a 14B, there's a- **Eugene** [37:49] Okay. So, so, so let's- **Host** [37:50] What are the assets or the- **Eugene** [37:51] Yeah **Host** [37:51] ... the artifacts. **Eugene** [37:52] Okay. So on a, um, okay, before we go into the nitty-gritties of how the layering and everything works, right, on the high level, right, currently RWKV architecturally as a model, it can be s- what we have already proven is that it can be scaled and trained like a transformer. **Host** [38:06] Mm-hmm. **Eugene** [38:07] Uh, how we do so, we'll cover later. And, and this, this can be scaled to as many parameters as we want. Currently, what we have is a, a dominant, uh, our main models is the 7B model and the 14B model- **Host** [38:19] Mm-hmm **Eugene** [38:20] ... which you can find on Hugging Face or respectively our demos. Uh, we also have an... There'll be the, there'll be the Rav- uh, RWKV Raven models. These are also instructionly tuned, uh, for... **Host** [38:35] It's not here. Okay. **Eugene** [38:36] Ah. I'm so sorry. **Host** [38:40] It's okay. Take- **Eugene** [38:40] And, uh, it's probably at the bottom, models. Down, down, down, down. It's the links. Models. **Host** [38:44] I see. **Eugene** [38:44] Yeah. **Host** [38:44] Okay. **Eugene** [38:45] It's on Hugging Face. These are the UX issues that I need to fix. **Host** [38:49] Yeah, yeah. You d- you only discover it when you talk it over with somebody. **Eugene** [38:51] Yeah, I know. So- **Host** [38:52] Okay, so there's World, there's Raven, there's Music. **Eugene** [38:54] Okay. **Host** [38:55] Oh my god, there's Novel. What, what is all this? **Eugene** [38:57] Okay. So, um, be- before we were... The, the current main models is RWKV for Rav- uh, the PAL and Raven. So this, so PAL is basically just a Pal Plus model. Um, it's- **Host** [39:13] What is Pal Plus? I know about PAL, but what is Pal Plus? **Eugene** [39:15] Uh, random data sets that they combine. The PAL. **Host** [39:19] How many tokens worth? **Eugene** [39:20] Um, we... I, I would just say slightly 1.1 or 1.2 times the PAL. **Host** [39:26] Okay. **Eugene** [39:26] Yeah. Um, this is not instruction tuned. **Host** [39:29] Okay. **Eugene** [39:30] And, and, and stuff. Yeah, the plus one is typically all the other languages. **Host** [39:35] Okay. **Eugene** [39:36] Um, subsequently, Raven are the instruction tune model. This is the current main complete models. We n- we subsequently have- **Host** [39:44] And the instruction data sets are from? **Eugene** [39:46] Uh, typically, uh, GPT-4. But then we scrub it for and remove all the- **Host** [39:53] SLR Transformer **Eugene** [39:53] ... SLR Transformer model. **Host** [39:53] So yeah, this would be the uncensored, uh... There's, there's someone, there's some other project that's kind of doing something similar, and they call it uncensored, but really they just scrubbed it as, as a large language model stuff. **Eugene** [40:03] Correct. Yeah. **Host** [40:03] Um, so that makes it technically breaking, uh, TOS of, uh, of Open, OpenAI, right? **Eugene** [40:11] Yeah. **Host** [40:11] Okay. But yeah. **Eugene** [40:12] But that's a... I mean- **Host** [40:14] That's a later problem **Eugene** [40:14] ... listen, frankly, let's be honest. If- **Host** [40:16] Yeah **Eugene** [40:17] ... if we, even if we don't remove it, someone is gonna remove it. **Host** [40:20] Well, I mean, so there's ways around this- **Eugene** [40:23] Yeah **Host** [40:23] ... which is you, you get it, you get clean data sets that are not GPT-4. Like, so the one that I typically mention is Yannic Kaiser's Open Assistant. **Eugene** [40:31] I, I believe that was included subsequently as well. **Host** [40:33] Yeah. Okay. Yeah. Obviously all these release orders are all over the place. **Eugene** [40:37] Yeah. **Host** [40:38] Uh, so, so okay, Raven, World, uh- **Eugene** [40:39] So Raven is the instruction tune model. **Host** [40:41] Yep. **Eugene** [40:41] The, uh, and then subsequently the World model is a new model that we are training. It's not 100% complete yet. **Host** [40:48] Okay. **Eugene** [40:48] Uh, with the focus on a new tokenizer and all the languages. So, s- so what we- **Host** [40:55] All the languages? **Eugene** [40:55] All the languages that we can grab from the internet. All the, all the wikis in all the respective languages. **Host** [41:01] Mm. **Eugene** [41:02] Uh, now th- this, please don't use five. Five was not yet ready. **Host** [41:06] Okay. Yeah, yeah. **Eugene** [41:06] Yeah, four. **Host** [41:06] No, no. I just wanna see the description, right? **Eugene** [41:08] Yeah. **Host** [41:08] Like, what do you mean when you say all languages? 100 languages. Okay, fine. **Eugene** [41:11] So 100 languages. Um, it wasn't really a very precise science. We just basically, whatever the, the, the wiki tool that allows us to download the X, X wiki languages, if it works, it's in the set. If it doesn't work, skip. **Host** [41:27] Yeah. **Eugene** [41:27] And, and you, and all the, uh, uh, and major prominent Oscar translation sets. So as you can see, PAL, Red Pajamas- **Host** [41:33] All right. What is Oscar? **Eugene** [41:35] Oh, Oscar, Oscar is just a common term that we use in... You can just search Oscar in Hugging Face dataset, and it just means translations. **Host** [41:41] Okay. **Eugene** [41:42] So you can find, like, English X pairs. **Host** [41:45] I see. **Eugene** [41:46] Yeah, all the respective pairs. **Host** [41:47] Okay. **Eugene** [41:47] Yeah. So, and then all training data I can find. **Host** [41:50] Okay. So 70% English, 15% multi-lang, 15% code. Is there, is there a strong groun- grounding for why 15% code? **Eugene** [41:58] Um, no. It, it was just, it was already there. **Host** [42:01] Yeah. So the- **Eugene** [42:02] The, the focus of the World model was not to improve everything else. **Host** [42:06] Okay. **Eugene** [42:06] It was literally that 15% multi-lang. We wanted to increase- **Host** [42:09] Right. It was, it was English and code, and then you just added multi-lang. **Eugene** [42:12] Yeah. We had a fair bit of multi-lang, but we wanted to bump it up, right. So we- **Host** [42:16] So this, this is primarily English? Whatever. Okay. **Eugene** [42:20] Yeah. **Host** [42:20] All right. Uh, what I would like is, like, basically like a visual of, like, here's all the building blocks and here's how they combine to, to create all these things. **Eugene** [42:27] Ah. So- **Host** [42:28] So **Eugene** [42:30] So we have the RWKV architecture code. **Host** [42:32] Yeah. **Eugene** [42:32] So that's the main model building block, and basically we feed it the data. Uh, PowerPlus, uh, RedPajama, then subsequently n- some that were code, code data. For the WoM model, we subsequently add on top of that, uh, all the translation mo- uh, Oscar sets and so on. **Host** [42:48] And so you're training these things. Uh, you've mentioned that you're intentionally taking a hit on evals, on traditional evals, like mMLU or whatever. **Eugene** [42:56] I wouldn't say intentionally. Uh, also to clarify, like I am not training it. **Host** [43:00] Yeah. **Eugene** [43:00] I'm just part of the community. **Host** [43:01] Of course. **Eugene** [43:01] The co- um, the community, and Blink is the one training it. But, um, I would say it's more of like the lack of care for the evals. So we-- the reason why we add things to the data set was never about improving evals. It's about directly in response to user feedback. **Host** [43:21] It's like, "Oh, not good enough at this." So they're, "Okay, just throw sh- throw shit in." **Eugene** [43:24] Yes. Literally. **Host** [43:25] Yeah. **Eugene** [43:25] Along those line. And it's also- **Host** [43:27] Interesting **Eugene** [43:27] ... so take, take for example, right? Like within, within-- for even for Raven and the WoM model, as we go through the training stages, right, uh, we, we specifically ask people in other nationalities within our co- Discord community to test it for their language. And our, our rule that we set is that, our informal rule is that the only person who can decide if whether this improved WoM model is better in Japanese or Thai or whatever it is, is a native speaker. **Host** [43:55] Yeah. Where, where does it take place? **Eugene** [43:58] So it's mostly in within, uh, linguistics sense, but sometimes we do a shout-out in general as well. **Host** [44:02] Okay, linguistics. **Eugene** [44:04] Yep. **Host** [44:05] So why don't-- So do you have like a appointed ambassador? Like you have 100 languages. **Eugene** [44:10] Yeah. **Host** [44:10] Do you j- just have like a, a tsar of Japanese, a tsar of Thai and- **Eugene** [44:15] It's not so appointed. It's more of like, "Hey, this is the Japanese model. Please try." It's, it's, it's not like- **Host** [44:23] But there's not the Japanese model. There's- **Eugene** [44:25] Oh **Host** [44:25] ... there's one model. There's, there's WoM model. **Eugene** [44:27] Um, so if you go to WoM model, I don't know whether it's in, inside here. No, four. Uh, sorry. **Host** [44:31] Sorry. **Eugene** [44:31] Five, five is-- We should never put five on top because five is ex- fully experimental. **Host** [44:36] Okay. **Eugene** [44:36] So under files and bu- files and versions. **Host** [44:38] I see. I see. I see. I see. **Eugene** [44:40] So- **Host** [44:40] I see **Eugene** [44:41] ... so there's, you see there's a Japanese specific tune. **Host** [44:43] Yeah. **Eugene** [44:43] Uh, a Chinese tune. **Host** [44:45] Chinese. Arabic. **Eugene** [44:46] Tune. Then for all the other smaller languages, we actually ask them from the base WoM model- **Host** [44:51] Yeah **Eugene** [44:52] ... a bit, uh, itself, so, so feedback on it. **Host** [44:55] Uh- **Eugene** [44:56] So we actually released previously like 10% train, 15%, 20%. Like as it goes through the stages, and then it's like, "Hey, is this working?" **Host** [45:03] Yeah. **Eugene** [45:03] "Is it regressing?" So it's like evals, but with hu- real, real humans. **Host** [45:08] Done by, done by real humans- **Eugene** [45:09] Yeah **Host** [45:09] ... and not, not systematically. Um, is there a reason that you release, you also re-- So you mentioned 7B, 14, 14B. Uh, but I see also 0.1B, 0.4B, 3B, 1.5B. Like what-- Is, is, is that useful for people or- **Eugene** [45:22] Yeah. So- **Host** [45:22] Is it just for researcher? **Eugene** [45:23] Uh, 0.1 and 0.4 is frankly more for research. But some people do try to make use of them. Nothing stopping them. Uh- **Host** [45:31] Well, I mean, it's, it's extra-- Like these are just different architectures, different dimensions. **Eugene** [45:36] Yeah. **Host** [45:37] So it's actually extra cost to you to provide these things. **Eugene** [45:40] Oh, but specifically for the WoM model, what ha-- because we are trying a new tokenizer. We are- **Host** [45:45] I see. **Eugene** [45:45] Uh, and, and the reason why we, we trying a new tokenizer is that, uh, as, as, as... I think I'm co- is that one thing that we found, or more like I found surprisingly frustrating in existing tokenizer was that it was very English-centric. **Host** [46:03] And the existing tokenizer you took from GPT-Neo? **Eugene** [46:05] Yeah. **Host** [46:06] Okay. S- and just to, uh, I need to backtrack a little bit just for people who are not following along. GPT-J was the original Luther reproduction of GPT-3. **Eugene** [46:13] Mm-hmm. **Host** [46:14] And then GPT-Neo was the bigger GPT-J. **Eugene** [46:18] Yeah. I do believe- **Host** [46:19] 0.20B, something like that. **Eugene** [46:20] Yeah, I do believe they have a 20B model. **Host** [46:22] Okay. Yeah. **Eugene** [46:23] And there's- **Host** [46:24] And then, and then- **Eugene** [46:25] There's, there's actually, uh, for do-- I mean, for those outside of the open source- **Host** [46:29] Yeah **Eugene** [46:30] ... um, uh, space, in particular for the transformer, uh, I think one thing significant for GPT-NeoX was that, uh, it was one of the major models that had everything fully documented and they, like why they make this change in the architecture and so on and so forth. **Host** [46:44] Mm. **Eugene** [46:44] And that became like a basically reference note for all other subsequent open source models, because they were the early ones that was, were like ma- doing, doing a good transformer model. **Host** [46:56] Yeah. **Eugene** [46:56] And at least for the in- for, for large language model. **Host** [47:01] So GPT-2 was actually open source. You didn't-- People didn't find that useful? **Eugene** [47:05] No, people do find, uh, do reference that as well, but it's like the code is there. **Host** [47:09] Uh-huh. **Eugene** [47:09] And why do you do this? **Host** [47:12] Oh, so it's not documented. I see. **Eugene** [47:14] Yeah. **Host** [47:14] I see. I see. **Eugene** [47:14] Yeah, yeah, yeah. **Host** [47:15] So in that sense, uh, was OPT, uh, uh, from Facebook useful? Because I, I've heard very good things about the logbook of OPT. **Eugene** [47:23] Mm-hmm. **Host** [47:24] Uh, where they, they had the daily logbook, and they just published that. **Eugene** [47:27] Yeah. Th- those were useful as well. **Host** [47:28] Yeah. Okay. **Eugene** [47:29] The, I think one thing that N- uh, NeoX had going for it, especially the Eleuther community, is that it, it's not just logbook. It's just like you could just go to Discord, "Hey, why you do this?" **Host** [47:40] Right. And the person who trained it will tell you. **Eugene** [47:43] Yep. Someone there hopefully can tell you. **Host** [47:45] Hopefully. Hopefully. **Eugene** [47:46] One of them. So that's why we had the 0.1 and 0.4 models because we, we were just in un- uncharted waters here. We-- So like a lot of existing tokenizer took space as a major delimiter to detect and split. **Host** [47:57] Mm-hmm. Mm. **Eugene** [47:58] And the tokenizer we are using is actually, um, a lot more simplified. So existing tokenizers, they, I mean, they scan all the text. They do a set this is model of like what, what, what pairs well with what and so on and so forth, right? Um, we did a similar approach, but we, instead of using like this token pairs well with this and should be paired with that, uh, we just made it a, a, a trie list. So, so basically we find the- **Host** [48:26] Trie, trie, T-I-E? **Eugene** [48:27] Yeah **Host** [48:27] The data structure? **Eugene** [48:28] Yeah. **Host** [48:28] Got it. **Eugene** [48:28] So we just find the longest matching string- **Host** [48:31] Uh-huh **Eugene** [48:32] ... in that, uh, matching string that we have trained inside our token list. **Host** [48:35] Uh-huh. **Eugene** [48:36] And then we just use that as a token. It's a drastically simplified tokenizer, um, and it doesn't use space as an assumption, which I know- **Host** [48:45] Which is good **Eugene** [48:45] ... yeah. And that helps a lot of the Japanese, Chinese, and- **Host** [48:49] Oh **Eugene** [48:49] ... character models because they don't have spaces. **Host** [48:51] Mm-hmm. **Eugene** [48:52] And I would even, and I, I would even argue to fair say, like if you look at like, like the really large models, like be it OpenAI or CloudAI, right? Tokenizers are not really a thing. I mean, the, in the sense that the model can work even if you tell it character by character. It's maybe inefficient. **Host** [49:12] Did someone try it? **Eugene** [49:13] I mean, there was that, that jailbreak where, you know, the system prompt, you put the character that enter, enter, enter the... You remember that jailbreak? **Host** [49:19] No, I didn't see that one. **Eugene** [49:20] Yeah. So, so you can literally, like instead of like left to right, you can do it up to down. **Host** [49:26] Okay. **Eugene** [49:27] And you're just eating tokens for, for every character. No, actually you're eating two because there's also the new line. Uh, and, and the model understood it because there's enough dumb data on the internet that it has learned how to deal with this kind of formatting. **Host** [49:43] Got it. Okay. **Eugene** [49:44] And if these models are already understanding things at the character level- **Host** [49:48] Okay **Eugene** [49:48] ... everything else is just improved compute. **Host** [49:51] Okay. **Eugene** [49:51] Because we, we jump the multiple tokens. **Host** [49:54] Do you have any idea of your dictionary size when you have, when you use this tree data structure? **Eugene** [49:58] Yeah. Um, I can- **Host** [50:00] 'Cause the, the typical tokenizer is like 80,000 tokens- ... uh, dictionary size. I presume your, yours will be bigger. **Eugene** [50:07] Yeah. It's aro- I can't remember offhand. Our, our previous tokenizer is around 50,000. Uh, uh, is the new X tokenizer, then subsequently, I believe this is around the same size. **Host** [50:17] It's not bad. **Eugene** [50:18] Yeah. **Host** [50:18] Pretty good. **Eugene** [50:18] We, we didn't wanna change too much, uh, on that size, but we just wanted to like, just change the format. **Host** [50:23] Mm-hmm. **Eugene** [50:24] Yeah. **Host** [50:25] Yeah. Cool. All right. Um, all right. What else should people know? **Eugene** [50:28] So, so world model is the, is the- **Host** [50:30] Ah, there's music. **Eugene** [50:32] Ah. No. You literally- **Host** [50:33] Sorry, sorry, sorry. Yeah **Eugene** [50:34] ... you, you literally just landed into like here's the experiment zone. **Host** [50:37] Let's talk about it. **Eugene** [50:38] Yeah. **Host** [50:38] This is cool. **Eugene** [50:39] So, um, so RWKV, fundamentally, i- it's, it's still an input-output model. **Host** [50:45] Right. **Eugene** [50:46] And you could do it for anything that you want. So the, so people, so there is actually another project internally on the Discord where it's doing vision, uh, vision modeling, uh, and this is based on the, the, uh, is it MiniGPT-4 paper? Where- **Host** [51:03] Okay **Eugene** [51:03] ... where you have an image model, put everything inside the latent space, and then you have the language model interact with that latent space, and then train both, and then you, you can do, do image stuff. Music was basically let's just take the same model, same code, and you know how MIDI files work, right? So the, like the MIDI files, just input and output MIDI files. **Host** [51:20] Okay. **Eugene** [51:21] And yeah. The, the, the... and there's actually a lot of other experiments, be it for vision. There's even an image generation experiment using RWKV. Uh, I'm not so sure whether it's in the list here. **Host** [51:32] There's clip gui- clip guided auto, autoencoder, but I don't think that's- **Eugene** [51:35] Yeah **Host** [51:35] ... the right one. **Eugene** [51:36] I wouldn't say it's a good image generator. Admittedly, but it worked. So, so- **Host** [51:42] So the, the... what I like about the transformers driven image generators is that they can do text well, and they can do control very well. So like you, if you ask for, you know, green, blue, red, uh, cars in, uh, arranged next to each other- **Eugene** [51:56] Mm-hmm **Host** [51:56] ... they will actually know how to follow that, whereas the diffusion models tend to treat it more as like a suggestion. **Eugene** [52:01] God, yeah. **Host** [52:02] You know what I mean? Or they'll combine the green, blue, and red into like one car. **Eugene** [52:05] Whatever it felt like it, right? **Host** [52:07] Yeah. You know, so, um, okay. But just to get back on this, um, okay, what, what, what else? Uh, yeah, so w- again, I actually kinda wanna establish the credentials of this thing. So who is Blink? Is it rando on the internet? Or like, because, uh, again, never heard of this guy until you published. **Eugene** [52:25] Bo Peng. Yeah. That, that's his o- real name. **Host** [52:26] Right. Um, and you had like... I have this paper to work with, but it was only published in May. **Eugene** [52:32] Yeah. **Host** [52:33] You found this before the paper. A- and so I think it's very unusual for a researcher to effectively launch, uh, to the au- to the wider public without a paper and just get some kind of pretty decent community going, um, and then publish the paper. **Eugene** [52:52] A- actually, it's the other way around. He- **Host** [52:53] Okay **Eugene** [52:54] ... got the basic community going before the paper. **Host** [52:57] That, that's what I'm saying. **Eugene** [52:58] Yeah. **Host** [52:58] This is unusual. **Eugene** [52:59] So, so, so the history, the, the history behind it, right, and is that, um, I, I, I think, I think like a few years back when, once tr- uh, ba- uh, with GPT-2, transformers started to pick up steam. **Host** [53:11] Mm-hmm. **Eugene** [53:12] And I guess the whole world is starting to think, "Let's just abandon neural networks." So, so we haven't even gone into the code part, but like, right, so the, the main reason why neural networks were bad compared to transformer was that when you, when you train a, let's say you just input a token and train a token for, for data sample, you have to wait for the compute to finish for that token, take the state, and then you train the next token. **Host** [53:34] Mm-hmm. **Eugene** [53:35] Uh, we'll get into how RWKV solves that. Uh, and but basically the whole world at that point just concluded, "Yeah, neural networks, it cannot scale as well. Transformer, let's just abandon it." And, and everyone just went in that direction. And Blink or Bo Peng is his actual name, uh, decided, uh, uh, basically as an individual, literally at the Eleuther AI forum, decided that, "Hey, um, I think we can modify recurrent neural networks, no, neural networks based on the Apple paper," the, the, the, the light engine that I showed previously, uh, to make, to scale this up without, uh, to make neural networks s- scalable and parallelizable in the same way transformers work. 'Cause the reason why we branch away and focus on transformer is because neural networks were slow to train. It was never, it, I mean, it wasn't so much about whether was it good or bad. It was just no one wants to wait 100 years for, for their billion tokens to train finish, even if they can throw a GPU farm at it. **Host** [54:36] Mm-hmm. **Eugene** [54:36] And, and that's where he started taking, looking into it, like how to make the neural networks trainable in parallel. And- **Host** [54:44] And, and specifically RNNs. **Eugene** [54:46] Yes. **Host** [54:46] Okay. **Eugene** [54:46] And subsequently, Eleuther AI and, and I believe there was also a few others, right, because he was doing it very publicly there, uh, came on board to sponsor, uh, the, the GPU computes required. Because even though it-- I mentioned that on large context size, it is substantially cheaper, I think, I think especially when-- if you're, if you run an open source Discord forum for an AI model, there every-- it's like every day there'll be someone who thinks that they can train a 20B model on a single GPU coming in. It's... The scale is still large. Even though it's like one-fifth of 110 compared to transformer, it still needs a large GPU. So, so that's why Eleuther AI and, and the rest, Stability, I believe also is involved, stepped up and donated the A100s needed to train the basic models that RWKV had. **Host** [55:37] Mm-hmm. **Eugene** [55:38] And, and at that... So before that, before those models were trained, right, we were like only having in theory like the toy models or the small models that this can match transformer. We have no idea whether it can match transformer at that- **Host** [55:51] Scale. **Eugene** [55:51] Yeah. **Host** [55:52] Yeah. **Eugene** [55:52] And subsequently, we-- the larger models, the 14B models and all that, that... And w-we can compare it directly with UX model, and that's where this paper came out. So, so that's the history behind it. It's like he wasn't really doing it in silence. He was doing it from Eleuther, then he branched out, uh, be-be, uh, because this became a big project on its own. ### Architecture **Host** [56:16] Mm. **Eugene** [56:17] And that's where other people started coming in as well. Um, so, so the part where we say that RWKV is a neural network that can be scaled-- can be rolled out as a transformer, right? **Host** [56:27] Yep. **Eugene** [56:27] The, the key thing that you want to, you would want to see, right, is this diagram here. **Host** [56:31] Yep. **Eugene** [56:31] Uh, this should be in the paper. Should... No, sorry. **Host** [56:34] Yeah, the other one. **Eugene** [56:37] Yeah. Accordingly. So what, what you get... So, um, when you do-- when inference, when you are running inference mode, ideally you should run it as a neural network. So this, this is a layer. So e-as per... So classic neural networks is that you have a state. The state could be start from blank. You, you process a token, you output a state, and then you rinse and repeat. And then as, as it keep doing the output, it, it makes a prediction, uh, in, in, in that. Uh, one thing that, uh... So subsequently for RWKV, what happens here, right, is that, uh, the-- we can roll out this neural network side by side, and then it, it runs another transformer. But the key thing here is that the states are split across the layer. So this is what we call... In this diagram here specifically, uh, this is what we call the time mix and channel mix. Uh, these are operations within, uh, the layer. Depending on how you want to view it, you could view this as individual layers, or as how we view it, we view like this collection of layers as one layer block. And each layer block pass the states to its sibling subsequently down the road as you process the next token. **Host** [57:46] Which is a similar RNN type- **Eugene** [57:48] Correct **Host** [57:48] ... feature. **Eugene** [57:49] However, the key thing is it-- you do not need to wait for the upper layers to complete- **Host** [57:54] Right **Eugene** [57:54] ... before you can go to the next token. **Host** [57:57] Mm-hmm. **Eugene** [57:58] So what happens in practice, and if I to jump to the diagram, like there's this graphic here. This is not 100% how it runs behind the scene. **Host** [58:08] I like it. **Eugene** [58:08] Yeah. But- **Host** [58:08] Whoever, whoever put time into this, uh, kudos. **Eugene** [58:12] I made it. **Host** [58:13] Makes sense. Makes sense. **Eugene** [58:13] Yeah. So, so, so, so, uh, th-so this is how you can visualize it. So this, the first layer is the layer norm. The layer norm doesn't... This is standard, uh, layer normalization that it doesn't need to... It just does, does it on token and doesn't need to wait for the other layers. But if you notice, right, subsequently to the right and to the top, these tokens, these blocks, right, need to wait for the blocks on the left. **Host** [58:34] Sure. **Eugene** [58:35] And, and this is like once you go past the first few tokens, right, this cascades very rapidly. Especially like this is only like one, two, three, four layers. So most models have like 20, 40 plus layers and, and the cascading patterns are happening. And in practice, once you start cascading there, you just saturate the GPU, and that's how it starts being parallelizable to train. You no longer need to train in slices like traditional RNNs. **Host** [59:01] Does big O notation help? Like, so we're talking about big O n squared- **Eugene** [59:07] Correct **Host** [59:07] ... for attention. Um, is this O of one or O of n? **Eugene** [59:12] Um, are you talking about like to, to go through the entire context? Yeah. Th-this will be, uh, O n. O one per token. **Host** [59:20] O one per token. O n for a whole sequence. Yeah. **Eugene** [59:22] Yeah. **Host** [59:22] Yeah. Okay. Cool. **Eugene** [59:24] Yeah. And- **Host** [59:25] And that's the core idea. **Eugene** [59:27] That was one of the key things. **Host** [59:28] What else is the key thing? **Eugene** [59:30] Uh, other th-- so other things is that, uh, so I think you, you're familiar with LSTM, right? **Host** [59:36] Mm-hmm. **Eugene** [59:36] Uh, this is, this is how like traditional neural networks keeps things within memories. **Host** [59:41] Mm-hmm. **Eugene** [59:41] Um, within, within here, within RWKV, we have two, we have two channels. So we, we call it the channel mix and the time mix respectively. Uh, uh- **Host** [59:51] And c- is there a formal definition of channel mix and time mix? **Eugene** [59:53] Yeah. Uh, we can actually scroll. But the def- like with this will be like going more- **Host** [59:59] Just formulas. **Eugene** [59:59] Yeah. **Host** [1:00:00] Okay. **Eugene** [1:00:00] Going more into the code. **Host** [1:00:01] Right. **Eugene** [1:00:01] Uh, so- **Host** [1:00:02] They're just weights or...? **Eugene** [1:00:04] Yeah, just weights. Uh, that, that, that applies according to the formula. But what, how it, how in a sense that it works, right, more importantly is you can see, you can see the data from the respective time mix and channel mix move to the next, uh, to the next segment. Um, ti-how time mix is designed, per se, was that, uh, it, it's how it retains. So like how LSD... Similar to LSTMs, right, where, where it processes state and the input, it may decide to discard certain states and keep new things in the state. Um, time mix does the same, same thing but with a different formula. So it's not... It replaces the LSTM- **Host** [1:00:41] Mm-hmm **Eugene** [1:00:42] ... in a sense. **Host** [1:00:42] Mm-hmm. **Eugene** [1:00:43] And, and it can decide to keep things indefinitely. So this represents the long-term memories if you want to view it that way. **Host** [1:00:48] But- Classically, the problem with that is that, uh, it struggles with long distance. **Eugene** [1:00:55] Correct. **Host** [1:00:57] Does it have the same issue? **Eugene** [1:00:58] So that's w- it's-- that's subsequent. Um, it struggles with long distance because it also needs to keep track of both near-term memory and long-term memory. **Host** [1:01:10] So you split it up. **Eugene** [1:01:11] Yeah. Re- effectively split up. So Channel Mix is, uh, subsequent- **Host** [1:01:13] The perf-- Is this the perfect memory? **Eugene** [1:01:15] Yeah. This is the closer to the perfect memory than is the short term. **Host** [1:01:19] Okay. **Eugene** [1:01:19] So, so, uh, else, uh, Time Mix, it, it has trainable weights on what it decides to keep and discard. **Host** [1:01:25] Okay. **Eugene** [1:01:25] Channel Mix, um, it has a very strong bias in it towards, like, just the next token. So, so you-- so, so subsequently it will just, like, as, uh, like memories are stored in the lower layers, it just slowly shifts upwards through the Channel Mix. And this is the short-term memory- **Host** [1:01:44] Mm-hmm **Eugene** [1:01:44] ... which at some point, as it just shifts all the way up, it will just disappear into the void. **Host** [1:01:49] Mm-hmm. Mm-hmm. **Eugene** [1:01:49] At that point, subsequently then Time Mix will-- should be retaining the longer term memory. **Host** [1:01:55] Are you also predicting-- Are you also sampling from a distribution? Um, so are you also sampling from a distribution? I-- So I noticed, for example, here that the illustrative demo is like, it says, you know, "My name is," and then it's, it's predicting, "Name is Bob." **Eugene** [1:02:08] Yeah. Correct. **Host** [1:02:09] Um- **Eugene** [1:02:09] That's our classic- **Host** [1:02:10] But is there, is there some amount of temperature, like a- **Eugene** [1:02:13] Oh, yeah **Host** [1:02:13] ... it's the same concepts that we- **Eugene** [1:02:14] Same concept. **Host** [1:02:14] Okay. **Eugene** [1:02:15] So it's the-- it's literally the same concept. **Host** [1:02:17] Log prob-probability of distribution- **Eugene** [1:02:19] Yeah **Host** [1:02:19] ... a-across your token space and yeah. Okay. **Eugene** [1:02:22] You could use Hugging Face sampler on top of it literally. **Host** [1:02:27] Okay. **Eugene** [1:02:27] So, so yeah, it-- the output is actually more of like a, a sell logic. Should we pause? **Host** [1:02:31] Pause for a bit. So we took a break for a bit, but now we're trying to cover, like, what is the big aha moment for you, and you said it was something to do with cost. **Eugene** [1:02:47] Correct. **Host** [1:02:47] Uh, so we have this chart on screen. It's literally a chart of quadratic scaling versus linear scaling- ... uh, in terms of GPU time spent in text generation. And you said it was, uh, at training time and at inference time. Like, just basically in everything that matters. **Eugene** [1:03:01] Correct. So, so I mean, so look back to, uh, how RNN works in, from a high level. We do an O1 operation on a token. **Host** [1:03:13] Okay. **Eugene** [1:03:13] Create a state. O1 operation, create a state. **Host** [1:03:15] Mm-hmm. **Eugene** [1:03:15] So, so this just scales linearly. You want to throw 1,000 tokens at it, it just, for on inference, it just scale linearly. Subsequently, uh, for transformer, um, it... you, you take in a token, you, you, you, you inf-- uh, you process your first token. It may be O1 here. Subsequently, when you generate your third token, you need to compute your second and first, and then vice versa. So if you do your 1,000 token, you need to compute back your 999 previous tokens. And as this keeps growing and growing, this is your quadratic scaling, and this is why, why we had th- we had this graph of the amount G- cumulative GPU time that you need to spend to generate all these tokens respectively. **Host** [1:03:56] Mm-hmm. **Eugene** [1:03:57] And this is fundamentally just transformer versus neural networks, um, yeah, on inference. Um, the reason why and, and subsequently, like, neural networks did have the disadvantage of, let's say, not being able to parallelize well in training. But as I, uh, as I covered, RWKV kind of solved that by splitting, effectively splitting the layers, allowing you to train different parts in parallel. Uh, and like some people will go into the academic debate of like, technically, the second and third token is not parallelizable until the first is done. But once you get into like, I can saturate a GPU then- **Host** [1:04:34] Yeah **Eugene** [1:04:35] ... it's just- **Host** [1:04:36] It's way better **Eugene** [1:04:36] ... it's, it's just academic debate. We are done. And so training, in essence, has always... I mean, this is paid for transformer. On neural network, it's I need to do an inference pass. I look at the logits. I then I backprop to see what went wrong, and I update the weights. Yeah. So the inference is the forward pass. You still need to-- It's part of the training course. Um, as you backprop as well, as you backprop as well, like having needing to only look at the current set of tokens and the state instead of, like, everything also reduce the amount of things that you need to backprop. So it's just that it's like there's so many factors involved in just reducing the overall inference and training time. And that was something that appealed to me because in the long run, I mean, all of us want our model to just run blazingly fast, right? **Host** [1:05:20] Yeah. Um, and also on minimal hardware. **Eugene** [1:05:24] Oh, yes. That. **Host** [1:05:25] Uh, which as far as I understand, you still have 14 billion parameters. That's not going away. You still need the, the RAM to store 14 billion parameters worth of stuff. That's not going away. **Eugene** [1:05:36] Yeah. **Host** [1:05:36] Okay. So RAM is unchanged. **Eugene** [1:05:39] Uh, yeah. On the RAM side, but the working memory is reduced. So typically, you need more than 14 for transformer. I mean, let's not, uh, let's not touch quantize, quantization. **Host** [1:05:51] Mm-hmm. **Eugene** [1:05:51] But in this case, we don't need to keep... Like, if you really, really want to, like, save RAM, you-- it is possible for you to do token by token inference so that you don't need to keep your states in history. **Host** [1:06:04] Okay. **Eugene** [1:06:04] You only need to keep your current token state and your next. **Host** [1:06:07] Yeah. Yeah. Okay. **Eugene** [1:06:08] And yeah, and, and that's actually like one segment of our community just purely porting other KB to C++ based model. **Host** [1:06:17] Or in the next. **Eugene** [1:06:18] Yeah. On a-- and running it on Pis and stuff. **Host** [1:06:21] Uh, Raspberry Pis. **Eugene** [1:06:22] Yeah. It's interesting to watch those. **Host** [1:06:24] Is, uh, is JAX interesting to people, TPUs? **Eugene** [1:06:27] Uh, there is some interest, but not- **Host** [1:06:30] People don't have access. **Eugene** [1:06:31] No. I would say frankly, the more-- the people with the most interest also happen to be the people who have free TPUs. **Host** [1:06:36] Yeah. **Eugene** [1:06:38] So I don't know. **Host** [1:06:40] My under- my understanding was Luther was also given a whole bunch of TPU hours, therefore they wrote their whole, all their stuff in JAX. **Eugene** [1:06:47] Yeah, and you-- if you can train it and then you've got the weights, you can always just run in something else. It doesn't matter, right? **Host** [1:06:51] Yeah. Yeah. Um, all right. Um, and then, and then there's a, there's a chart about, uh, performance, uh, and it shows that RWKV is competitive or, or actually better in some of the reasoning challenges, which that's something I definitely would look for, right? Like, and it's, it's fine if, like, your, your speed is faster and all that, but if the reasoning quality sucks, then it's not a very useful language model. **Eugene** [1:07:16] Exactly. **Host** [1:07:17] So- **Eugene** [1:07:18] So this is like literally us saying there's no- **Host** [1:07:21] No trade-offs. **Eugene** [1:07:21] Yeah. You don't lose out in that process. **Host** [1:07:23] Okay. Big question then. Why isn't R- RWKV a bigger deal right now? **Eugene** [1:07:29] Um, so- ... one, we are not a commercial organization. ### Obscurity **Host** [1:07:34] Okay. **Eugene** [1:07:35] This is literally the pure open source play. Um, so- **Host** [1:07:39] But you could have done the Stable Diffusion thing, which, you know, Stable Diffusion launched, um, it was by a bunch of nobodies before that. **Eugene** [1:07:49] Mm-hmm. **Host** [1:07:49] It's from, like, literally split out from Luther. Um, and but they definitely had some hype. They had, they definitely... Like, you know, I, I interviewed Sharif Shameem, who was, who got in, like, and I... This is something I, I... The, the reason I ask you so many things about how did you find out about RWKV, 'cause I think the, the generalizable skill is how to be early in AI. **Eugene** [1:08:09] Ah. **Host** [1:08:10] 'Cause ear- being early in AI is very valuable, 'cause then you were there to see the, the, how things developed in- instead of, like, picking it up later like me. Um, anyway, so yeah, why is it not a bigger deal? **Eugene** [1:08:23] You know, to be frank- **Host** [1:08:24] Yeah **Eugene** [1:08:25] ... we just suck at marketing. **Host** [1:08:26] Okay. That's fair. **Eugene** [1:08:28] But I mean, uh, uh- **Host** [1:08:29] This is part of it. **Eugene** [1:08:30] Yeah, this is part of it. Like, um, so, like, maybe- **Host** [1:08:34] But, like, a- again, like, I don't think that is entirely the, the cause. **Eugene** [1:08:38] Yeah, I'm sure. Definitely. I think the other major segment right now as well is that, uh, is that we were really late on the paper. **Host** [1:08:46] Okay. **Eugene** [1:08:47] Like, one of the weirdest thing right now is I, I, uh, weirdest thing right now I feel that is that RWKV is starting to have its moment right now. **Host** [1:08:55] Okay. **Eugene** [1:08:56] Is that ever since that initial paper came out, there was ResNet. There's, uh, um, I think there's two more, uh, uh, there's a few more additional papers coming out, one from Microsoft, one, uh, one from other organizations, that are literally exploring the whole idea once again of scalable neural networks. **Host** [1:09:13] Okay. **Eugene** [1:09:14] And they are citing RWKV as part of it as well. **Host** [1:09:16] Okay. **Eugene** [1:09:16] And, and I think for most, uh, almost like I think it's existingly, why switch to this model when even though we have proven that, yes, it's scalable to 7 and 14, and that it can match transformers at similar param and training size, but all this is at very academic. Because the co- the, the community, right, uh, the comm- the community at large, especially for the English commun- uh, speaking community, right, they don't really care about this. They care about what's the best model that I can run on my computer, at least within the open source space. And by that, and even though we match in performance for things in the same dataset, the keyword is same dataset. Like, uh, this benchmark is not, it's not even Red Pajamas yet. It's, it's the pal. **Host** [1:10:09] Mm-hmm. **Eugene** [1:10:10] And, and when you have models that are, like, a bit like Falcon being trained on much larger dataset- **Host** [1:10:15] Mm-hmm **Eugene** [1:10:15] ... for, especially for an English use case, it makes more sense to use that. **Host** [1:10:20] I see. So there, there will be another paper coming that is RWKV trained on Red Pajama- **Eugene** [1:10:26] And- **Host** [1:10:26] ... that will presumably be a larger- **Eugene** [1:10:27] Or larger dataset. Yeah. **Host** [1:10:28] Okay. **Eugene** [1:10:28] And so on, so forth. So, so I think that's the, we are still in the stages of reaching that point where we train on the larger dataset. **Host** [1:10:35] Yeah. **Eugene** [1:10:35] The only reason why we have a bigger outsize impact compared to, like, the other models is frankly because half of our Discord came in not for English. It's for other languages. **Host** [1:10:47] Yeah. That's great. Um, and there is a definite very US and English-centric bias towards, um, uh, these models and, and it's, it's, to me, kind of poetic. Like, there's nothing in, in the architecture of R- RWKV that particularly bias it to be really good at other languages. **Eugene** [1:11:06] No. **Host** [1:11:06] It's just that as a community, you decided to prioritize it- **Eugene** [1:11:09] Yeah **Host** [1:11:09] ... in your tokenization, in your datasets. That's it. **Eugene** [1:11:12] Yeah. That's it. I, I, I, um, I would even argue that I'm surprised, more surprised that especially on, on the European side of s- things, that we don't have more, more m- more models that actually focus on even the European languages because there is, like, a so- softer jump to character- **Host** [1:11:33] Yeah **Eugene** [1:11:33] ... Japanese and Chinese character. **Host** [1:11:34] Right, because they're all romantic. I would say, well, one, Europeans are very hostile to tech advancement, you know? The first thing they, they have never met a technology they don't, they cannot regulate, right? Everything is just ban, regulate, ban. Um, whereas, um, you know, and then on, on our side and the Asians like to, like to have waifus and so that would be my, my guess. Um- **Eugene** [1:11:56] Yeah. But I think back to the benchmark, what excites me most still about this is that it just means that we just need to scale. We just need to scale this model and with the right data- **Host** [1:12:07] To, like, 40B? **Eugene** [1:12:09] 40B, 60B. **Host** [1:12:11] Yeah. **Eugene** [1:12:11] The, uh, I mean, it's, params is one thing. It's datasets- **Host** [1:12:14] Okay **Eugene** [1:12:15] ... and GPU time. **Host** [1:12:15] Yeah. So you and I have talked offline about ideas for getting data, getting compute- **Eugene** [1:12:20] Mm **Host** [1:12:20] ... and all this. Okay. Um, so this is, like, a project that's, that's o- is ongoing. Um, okay, anything else for the future of RW- RWKV? **Eugene** [1:12:29] Uh, the biggest one would be... Okay, so, um, this is back to how remember I said evals doesn't hide or, uh, doesn't highlight everything. Like, like, like, like this is nice and all, the evals, but there's, being realistic on another weakness on the RWKV side is that, um, now with the rise of like, let's say, 100K or 32K context size windows, uh, trans- uh, transformer model, RWKV currently is trained To handle, let's say eight or even some people have already trained it to 16K- ### Scaling **Host** [1:13:01] Yeah **Eugene** [1:13:02] ... sizes. Um, it has-- And well, it will, as a neural network, it will happily keep going on for infinite context length. It will just keep generating. Uh, does it do well? That's the th- the, the answer is no, because if you didn't train it to handle that situation, and there's actually a chart below. So for example, if like the prediction, the, the PAL test loss, right? It does improve over time, let's say, if you go down the context length. But this is if we train it. And what is not seen here is that if we were to, do, do, let's say, run it further, it would just go back up. **Host** [1:13:34] Yeah. **Eugene** [1:13:34] Because it was not trained to handle that. Uh, well, it technically can run. It, it suffers from the longer context length. And that, that's the part where RWKV, especially in like Q&A task, in huge documents, like- **Host** [1:13:50] Yeah **Eugene** [1:13:50] ... you get closer to summarize giant documents. That's where it starts to- **Host** [1:13:53] Look, like- **Eugene** [1:13:54] ... fall **Host** [1:13:54] ... none of this is fundamental. It's just you need more money. **Eugene** [1:13:57] Yeah. **Host** [1:13:57] That's it. **Eugene** [1:13:58] And I, uh... No, there is actually a fundamental part. So, uh, what, what one of the things that I was doing, uh, ac- uh, am actively helping within the community right now, uh, is that we found that, uh, the existing way to scale the memory was not that efficient, and we were just being realistic ourselves. If we want to hit 100K, we need to change this. So, so one thing, one thing that I'm actually looking forward to right now is actually those experiments. Uh, we have already s- we have already started scaling, scaling things to be able to, to handle things at transformer scale, uh, be it the 4K, 8K, in terms of like how it handles memory really well. And we want-- And we found it-- We want to like extend that to be like 16, 32 and 64. And we, we, that is within our roadmap, and that's the exciting thing. Because once we have that, like it able to handle long-term memory within those sizes, uh, it remove what many people in the community felt, right, was the last architectural limit. **Host** [1:14:57] Mm. **Eugene** [1:14:57] Because once it's able to handle memories like context length with tr- the same as transformer, we no longer need to do all the... Like, you know how existingly, like people do like long conversation in transformer, they just discard the rest, and the sliding window. This is like the better version of sliding window. You have-- The model can handle the sliding window perfectly, where it can keep remnants behind it. **Host** [1:15:20] Sure. **Eugene** [1:15:21] And that's something that I'm really excited and invested towards because this is back to the full circle, what, how I came into RWKV. I want my model to handle 100K tokens, four megabytes of HTML, whatever I throw at it, and be able to process it. **Host** [1:15:38] But it'll be, it'll be lossy. **Eugene** [1:15:40] The, the latter half will be lossy, but the key thing is extending the non-lossy part, and we are aiming to extend the non-lossy part. **Host** [1:15:49] Okay. Interesting. Um, great. Uh, that was, that was really good. Um, oh, one thing I wanted to cover before we leave the topic of RWKV, uh, altogether. Uh, th- there's a, there's a couple things, but first of all, what is it like working-- Basically, it's a vol- all volunteer Discord anonymous community. You've never met any of these other people. Um, which is like, it's only been done one other time successfully, which is Eleuther, right? **Eugene** [1:16:15] Correct. **Host** [1:16:15] In a way, RWKV is kind of new Eleuther. Ob- obviously, Eleuther, Eleuther is still going, but in, in, in as far as active research in like something that's completely untested by like complete nobodies, it's you guys. How-- What is it like to organize a group like this? ### Community **Host** [1:16:33] Ex- I've never, I've never organ- I've never been involved in something like this before. It's so weird. **Eugene** [1:16:38] Yeah. You-- When we use the word organize, it makes it sound like, uh, there is more organization than it actually is. **Host** [1:16:43] I would like... Yeah. If I think about how I, how I've typically organized-- done projects, I would like try to assign roles or try to have regular meetings or, you know, like try to have some... Everyone is volunteers. **Eugene** [1:16:53] Yeah. **Host** [1:16:53] Nobody has any like w- means to order people around or anything like that. But like how do you collaborate if you don't know what each other are doing and don't have people that like are not committing to deadlines or, you know, like do you have a Jira board? Do you have any... **Eugene** [1:17:07] Okay. **Host** [1:17:07] Um, so, so going-- bringing it back to the Discord. Um, uh, so Blink is a busy person. You are, you are definitely very involved in the Discord community- **Eugene** [1:17:15] Mm **Host** [1:17:15] ... organizing. How do you get stuff done? **Eugene** [1:17:17] So, uh, so I think Blink, the one who-- Blink is also the one who has access to the main EleutherAI, uh, and Stability, Stability's GPU donations. **Host** [1:17:28] Okay. **Eugene** [1:17:29] So he's the one that is very focused on training the big foundation models. And, and that's, and that's what he do. So right now, right now in our current pipeline, he's focusing on the WoM model and subsequently some experimental models for RWKV-5, which is the, the next generation. And the WoM model is like, is our next major foundation model when it's fully trained. Uh, it, it will cover all the other languages. And from there onwards, he just generally continuously keep the Discord updated on the progress of it, be like, "How is it going? Where is it going?" Um, he's, uh, he constantly highlights the, the, the projects that are being submitted, and the internet is just now, eh. **Host** [1:18:12] Oh, yeah. I've been tethering the whole time. **Eugene** [1:18:14] Oh. **Host** [1:18:14] But that's okay. **Eugene** [1:18:14] It's okay. Uh, and then subsequently he updates with his ideas and his plans and so on. Like there's even ideas, as you can see for- **Host** [1:18:20] It's pretty cool. Yeah. For V6 **Eugene** [1:18:21] ... for V6. Yeah. There's like long-term, several... But these are like big ideas and sometimes, uh, in a lot of times, right, he's very focused on the text models and also some of these ideas need to be tested and validated. So this is where, where things starts branching off per se. So, so for example, uh, one area that I s- I started being active in, in was that I was... At first I was-- At-- When I came in, at first I was being more active in, let's say the packaging, the inference code to make it more accessible. So I think one of the things that I showed was the RWKV Node.js module. **Host** [1:19:01] I can see it. **Eugene** [1:19:02] Yeah, it's fair enough. The Node.js package where basically you can run RWKV in Node just, just to make it more accessible. And, and then subsequently I was supporting that. And as m- then as more people came on board, like trying to run it in their respective languages, uh, I subse- Ah, it's okay. **Host** [1:19:19] It's working. **Eugene** [1:19:19] I'm just gonna keep going. **Host** [1:19:20] Yeah. **Eugene** [1:19:21] Uh, I subsequently moved on to focusing more towards datasets and RWKV5. But this is the area that I'm focusing on and most active, and this is how we start organizing as well. Like individuals have generally have their own area of focus- **Host** [1:19:37] Mm-hmm **Eugene** [1:19:37] ... of what they want, and it's very focus driven on s- in a lot of cases aligned to them. So for example, like the infer-- the people who are working on inference, uh, the CPP mo- the, the, the, the ONNX model, the CPP v-versions, right, where it takes the existing model and converts it accordingly. They are highly motivated to do this because they want to do the inference in their use cases, in their Raspberry Pis, et cetera. Um, people like me who's in RWKV5, we are actually more of like we know there are some weaknesses in the model, and we are trying to make those changes to improve. So we are like actively changing the, the, the foundation code. Then from there onwards, there are like channels. So these are the RWKV5 channels. Uh, I mentioned the, the, the inference channels, the, the CPP channels. And then from subsequently, there is also the multi-model channel. So, and this is an area where I am not fully active in, but there are individuals who are very interested in like getting visual recognition, uh, mini GPT-4 audio. Um, apparently the music thing is catching with up within the community right now. People are getting excited, uh, pe- about it. But th- this is where it- various other individuals come in to just contribute to that side. And this is still within the sphere of like code and engineering. And like if I go subsequently back down another step, there is also the multi-language channel and the dataset channel, and this is where it, where you find individuals who are just, I would call, I wouldn't say they are like playing the role of librarians, who's just trying to like find the right datasets, label it, collate it, clean it up, and then put it in part of the training. And, and their, and their typical focus is that they want to s-support their language better, or they have their, I guess, like you alluded, their waifu use case. They want to make it do better. And a- and that's how the community driven effort is done because everyone actually has a certain incentive and alignment, and they just double down on it effectively, and they start to take a heavy active role in the channel. So like frankly, I'm not gonna say that I'm active in multi-model because that's an area where I'm not really active in. **Host** [1:21:49] Yeah. **Eugene** [1:21:50] And so, and, and that's how we try to like self-organize, and we share our notes accordingly. Uh, we sometimes just casually just hang out on the Discord voice chat or whatever, and then we just talk with each other. But that's more of like the more, the more casual stuff of it. But how things get done, it's, it's down to the individual groups. **Host** [1:22:10] Has Beau stated his ultimate end goal apart from this is cool? **Eugene** [1:22:16] Mm, I think we had several co- I had several Discord, uh, conver-conversations on w-with him. Uh, I believe that what he... Because I did ask him frankly, like, is he planning to make a commercial entity out of it? Actually, tons of peoples have asked this because that seems to be the pattern. And he seems to be heavily inspired and wants to go towards the direction of creating the equivalent of a Linux Foundation but for AI models. So he really wants this to be open source. **Host** [1:22:43] Mm. **Eugene** [1:22:43] Um, and that's actually part, part of what motivates me to just continue on in the Discord as well. **Host** [1:22:49] Yeah. Yeah, yeah. Um, do you, do you think that has, has-- is that a serious effort? Uh, because that, uh, I, I might be also looking to explore. I, I know some friends who are also working on like an agent protocol- **Eugene** [1:23:03] Mm **Host** [1:23:03] ... that could benefit from a neutral nonprofit foundation type thing. **Eugene** [1:23:07] Mm-hmm. **Host** [1:23:07] Um, so we might want to w- we might wanna work together to set it up. **Eugene** [1:23:11] Yeah, sure. Because, um, I, I did post to him a few times, like, like we should at some point organize and set up the actual foundation- **Host** [1:23:21] Yeah **Eugene** [1:23:21] ... rather than the informal- **Host** [1:23:23] I think I know the people that, who would be able to help. **Eugene** [1:23:26] Yeah. That would be great because, uh, I mean, like I think for us, setting up the foundation will probably be one big major step because then it will also simplify the process in terms of like being able to handle GPU donations and stuff like that. **Host** [1:23:41] Yes. Oh, that's a good point. That's a good point. **Eugene** [1:23:44] Because right now a lot of the donations- **Host** [1:23:48] Well, so I saw that there is an RWKV Foundation. **Eugene** [1:23:52] Oh, no, it doesn't fully exist yet. **Host** [1:23:53] Oh, okay. 'Cause he listed himself in the paper as- **Eugene** [1:23:56] The, the, this is back to like the paper. The paper require you to list an organization that you belong to, and if you don't have a f-organization, what do you put? **Host** [1:24:04] Okay. Interesting. **Eugene** [1:24:05] So that's... We it's like, okay, he-- at some point we, we will need to set that up, so he just went ahead and filled it out. **Host** [1:24:12] Yeah. Yeah. Cool. Um, I think that's the RWKV portion is unless there's, uh, any other parting notes. **Eugene** [1:24:18] Mm. Yeah. Um, the, the, the Discord is filled with people always trying to do other many things. **Host** [1:24:25] Yeah. **Eugene** [1:24:25] If anyone has any interest in a really specific task, go ahead, join in. If you just want-- If you are from a foreign country that it seems like no model seems to care about your language, please do join in because we want, we want these people. We want to support your language, and we want to know how good or how bad our model is in that language. **Host** [1:24:43] So what I would do here as a product manager is like put up a public repo somewhere of like, "Here's all the language we wanna target. Here's our completion ratio." Like, you know, check, check, check, check, check. Blink, blink, blink. You know. **Eugene** [1:24:54] We need someone to do that. **Host** [1:24:55] Exactly. This, this would be a classic PM, uh, type of thing. Uh, but anyway, so anyone listening, uh, if you are interested, uh, Eugene is Pico creator. **Eugene** [1:25:05] Yep. **Host** [1:25:05] Um, you, you seem to be all over the, the Discord, so it should be pretty easy to find you. **Eugene** [1:25:10] Yeah. **Host** [1:25:11] Um, okay. And so that, that's basically the RWKV portion. Uh, you had one more comment, comment about alternative models, and you, you mentioned that you actually, apart from other, other RWKV, which is one thing, it's not like your whole identity- **Eugene** [1:25:25] Yeah **Host** [1:25:25] ... that you're very, you're very involved right now. You said that there's also po-potentials for diffusion models in text. **Eugene** [1:25:31] Oh, yeah. So, um, I think, I think for, for me the, the key principle is that we want to make sure we avoid the trap into landing on that one model to rule them all. ### Diffusion **Host** [1:25:42] Mm-hmm. **Eugene** [1:25:43] Because all models will at some point, from an architecture point of view, may do some trade-off. And if, let's say, we, we go back to the point where may- maybe all we need is a scalable model and a good data set, it's in the community best interest or more like the whole world's best interest because we are burning a lot of GPU energy and time to find an efficient model for, for all the respective use case. **Host** [1:26:05] Okay. **Eugene** [1:26:06] And, and, and, and all these are all trade-offs. So, so even if, let's say, uh, like fast-forward, maybe RWKV became the prominent model, I would still say that we need to explore alternative models because all models will have its weaknesses. So ROK-- one of RWKVs and Transformers models' weakness is that, uh, and I think there was a paper that covered, is the mo- is the multi-epoch and, uh, uh, how it can only tr- uh, how training you should ideally train from one to two epoch, and then- **Host** [1:26:35] Yeah, that's Aaron Kotomarski or whatever his name is. **Eugene** [1:26:37] Yeah. I can't remember off my head. Sorry. **Host** [1:26:39] Yeah, yeah. **Eugene** [1:26:40] Yeah. And, and- **Host** [1:26:40] Yeah. So his, his paper is literally titled One, One Epoch Is All You Need. **Eugene** [1:26:43] Correct. **Host** [1:26:44] I actually have observed that this is, uh, strange to me that you only train on one epoch for, for a whole data set. **Eugene** [1:26:51] Yeah. And anything beyond that... And we can confirm even for our model, ours is more like closer to two, but the idea is still there, that it will start to overfit and it starts to degrade in, in a lot of things. **Host** [1:27:03] Mm-hmm. **Eugene** [1:27:04] And, and I think this is a serious enough problem that, that within the transformer community, that we sometimes joke about the, the token crisis. **Host** [1:27:13] Yes. **Eugene** [1:27:13] That eventually you run out of tokens to train. **Host** [1:27:15] Do you think there's a token crisis? **Eugene** [1:27:16] Uh, I would say if we are aiming for AGI, there is a token crisis. **Host** [1:27:21] Mm-hmm. **Eugene** [1:27:21] But if we are aiming for useful small models, I don't think there is a token crisis. **Host** [1:27:28] Right. **Eugene** [1:27:29] 'Cause- **Host** [1:27:29] Um, let's say-- Let's talk about AGI 'cause I, I, I-- the small model stuff is, is, uh, I think a given at this point. Um, but right now, let's say, you know, uh, Llama 2 was trained on 2 trillion tokens. **Eugene** [1:27:41] Mm-hmm. **Host** [1:27:42] Can we go to 20 trillion? Can we go to 200 trillion? You know, like, uh, is there orders of magnitude left, or are we, like, basically almost done? **Eugene** [1:27:49] I think the one, one thing amazing with the Llama paper is that it showed that, uh, even at 2 trillion- **Host** [1:27:54] It's not leveling off. **Eugene** [1:27:55] Yeah, it's not leveling. **Host** [1:27:56] The models, the models are still going. Yeah. **Eugene** [1:27:57] So you could potentially train it for 16 or whatever it is. We don't know what's the limit. But the problem here is: Where are we gonna get the tokens? Because we already established that, that it's equally important that you have good data- **Host** [1:28:07] Quality tokens **Eugene** [1:28:08] ... yeah, that goes in, uh, rather than junk data. And- **Host** [1:28:11] Yeah **Eugene** [1:28:11] ... and that's the crisis, for lack of a better word. Um, and I feel that it might actually get worse mostly because, well, yeah, we can keep crawling the internet, but now with AI models dumping content to the internet- **Host** [1:28:25] Mm-hmm **Eugene** [1:28:25] ... you actually need to figure out what is quality content, and you need to start filtering. So this is literally like a librarian's job. Plus, like, we literally have, uh, we-- uh, one of the things that we explore within our company is, like, starting to classify our models. No, I mean, our datasets. Literally taking the, the library classification and- **Host** [1:28:43] Yeah, the Dewey Decimal System. **Eugene** [1:28:45] Yeah. **Host** [1:28:45] Yeah. **Eugene** [1:28:45] Uh, and then using that accordingly because there's just so much things. And, and as long as we-- Currently, one of the biggest gap that we, we notice is that while there are a lot of books, a lot of them are stored digitally as images. So in terms of text, there is actually a shortage. **Host** [1:29:05] Okay. Run a OCR step. I don't know. **Eugene** [1:29:07] Yeah. Easy to say. **Host** [1:29:08] Easy said than done. Easy said than done. **Eugene** [1:29:11] And, and, and that's where the token crisis lie. But, I mean, this diverging, uh-- this is back to why I'm interested in alternative. Because the reason why I pointed out diffusion model is that, yeah, transformer and, uh, large language models right now are having that one, two epoch limitation, and you go talk to people in the image space and they're like- ... "What?" **Host** [1:29:31] 50 epochs. Okay. **Eugene** [1:29:33] We, we s- 50 epoch is low. We, we do 200, 250. **Host** [1:29:37] Yeah. **Eugene** [1:29:38] And, and there, and there, there's various reasons for it. I mean, this is pure speculative. Uh, my speculative reason for it is that diffusion models work so well with, with multiple epoch because each training epoch, right, it is randomized with noise. And effectively, each training run, even if it's the same data sample, it is different due to the change the noise introduce or whatever is truncated and removed. And, and that's why it held up well. I mean, and if that is the case, if-- shouldn't we be exploring more, uh, as well into diffusion models, even for text, into emulating parts of this behavior or exploring... As I said, like, you, uh-- like, one of the reasons why diffusion models are not being used for text is because it's slow. **Host** [1:30:25] Uh-huh. **Eugene** [1:30:26] Shouldn't we-- alternative, could we be exploring how to make it faster? And this is why I feel like, like even from-- even when we talk about RWKV being, having the trade-off, yes, it's faster, it's scalable and whatsoever, there is other trade-offs that is still limited. It still suffers from the multi-epoch problem, and diffusion models may actually represent a, a potential for us to escape this token crisis and maybe train on our data set 200, 500 times. **Host** [1:30:54] That's interesting. I, I don't know how to respond to that apart from, like, I think it's a new perspective I haven't heard. **Eugene** [1:31:00] Yeah. But, uh, to be clear- **Host** [1:31:01] Yeah **Eugene** [1:31:01] ... this is all net speculation theory- **Host** [1:31:03] Net theory. Yeah, yeah **Eugene** [1:31:04] ... and I could be completely wrong. **Host** [1:31:05] Well, I-- It's-- Okay. You know, so to me, the, the speed thing really does matter. Uh, and being able to stream token by token actually is a g- is known to be good UX. **Eugene** [1:31:14] Yeah. **Host** [1:31:15] Right? And I'm not gonna wait for my, my essay to, like, slowly materialize- **Eugene** [1:31:20] Yes **Host** [1:31:20] ... from the diffusion process, right? Um, maybe-- But maybe you'll find some use cases that- **Eugene** [1:31:25] Or maybe we can just extract the part- **Host** [1:31:27] Right **Eugene** [1:31:27] ... where, where it's trained with noise and somehow survive multi-epoch. **Host** [1:31:30] Right. Um, and then the other criticism, uh, off the, off the top of my head of what you're saying is that like, you know, even RWKV and, uh, typical transformer models would have random initializations. So why can't we ju- if, if your thesis is that, um, starting from random init- initializations gives you the ability to do multi-epoch, like- **Eugene** [1:31:51] It's not so mu- diffusion is not just random initialization. It's... There is randomness in the data that they intentionally put in and as they remove in training. **Host** [1:32:03] Mm-hmm. **Eugene** [1:32:04] So it's not just at the s- at the start- **Host** [1:32:05] I see. As far, as far as the training process, they, they actually- **Eugene** [1:32:07] In the middle of the image **Host** [1:32:09] ... I see. I see. Right, right. Yeah, makes sense. **Eugene** [1:32:10] Yeah. **Host** [1:32:10] Yeah. Okay. Got it. Got it. Got it. Um, yeah. **Eugene** [1:32:12] How, how we translate that into a transformer prediction training, I have no idea. **Host** [1:32:18] Yeah. I mean, so, so my, you know, analogy would be they should make a Frankenstein R- ... RWKVD that just has some weird thing diffusion kinda slapped onto it, and then you're fine, you know? So- **Eugene** [1:32:30] And then maybe it proves that it's yes or maybe it just goes wrong. And, and, and I'm all for it. Like, someone needs to try. ### Advice **Host** [1:32:35] Yeah, yeah. Someone needs to try it. Okay, cool. Um, so we're gonna wrap up with just your ex- uh, so you know, you have displayed today an impressive amount of knowledge just across the, you know, all, all this stuff, and you're, you don't have, uh, like a research background. **Eugene** [1:32:50] No. **Host** [1:32:50] Um, your advice to AI engineers getting as deep as you, who want to get, get as deep as you. What, you know, any thoughts? **Eugene** [1:32:57] So I think your article articulated very well that there are, there's gonna be divisions within how we approach this. So AI engineers, um, mo- I, I-- sorry if I don't quote it correctly. **Host** [1:33:10] Yeah, it's okay. **Eugene** [1:33:11] AI engineers then in my head, the next level- **Host** [1:33:13] Well, the beau- the beauty of it is that s- I define the two words, and then everyone has their own definition, but they all roughly project onto the same embedding space. **Eugene** [1:33:23] Okay. **Host** [1:33:23] It's, it's beautiful. **Eugene** [1:33:24] Yeah. **Host** [1:33:24] It's... Yeah. **Eugene** [1:33:25] So AI engineers, model trainers, and dataset curators, and, um, uh, ML scientists, so I'll loosely, loosely define as a tree. I ignore the full stack because every company needs it. So, um, within this tree space, uh, there is actually a lot of ways anyone can come in without knowing anything. So let's just start with AI engineers. Um, don't be... Like, even though this, th- this whole topic, we even dived into how the layers work. We also showed how the math works. Frankly, for an AI engineer- **Host** [1:33:58] Yeah, not required **Eugene** [1:33:58] ... you don't need it. Um, you, your, your main th- uh, your main thing that you needed to do was to, frankly, just play around with ChatGPT or, or the alternatives. Be aware of the alternatives. Just be very mercenary. Swap out to Cloud if it's better for you, or swap out to an open source if it's better for you, and just play around with prompts. Learn, learn be- uh, prompting techniques like one-shot, two-shot, few-shots. Um, and, and then from there onwards, you can start building your agents, stacking, stacking your prompts and in sequences and stuff like that. And you are able to build application that do an- anything in, in terms of the AI space. And all this without n- knowing all this nerdy stuff w- uh, or the en- hard engineering, 'cause that's all you really need to actually build a product for the user. Remember, you are supposed to focus on making it for the user. They don't care if it's RWKV or transformer underneath the hood. They just care that it helps them. And, and I would say like Motion probably is like probably one good example of how they use it, because we know underneath the hood it's OpenAI, but it's... You really use- **Host** [1:35:07] It's great, right? **Eugene** [1:35:08] Yeah. **Host** [1:35:08] Yeah. Uh, no, but so I, uh, obviously agree with all that. Let's just say that people are there already, and they're just curious. They, they wanna do what you did. **Eugene** [1:35:17] Oh, so, so that's where you start going down the layers. **Host** [1:35:20] Yes. **Eugene** [1:35:20] So the, the next layer, the next layer w- you go down in is, is subsequently training the model, uh, be it from scratch, fine-tuning, and curating the dataset. And this is, this, this is from-- this is where you still do not need to know the math, but you need to know, like, you need to have a rough sensing on how the model works and how the certain models... And in, in this, even within the transform- open source transform space, certain models are better trained in certain sequences with certain learning rates, and you just need to get a feel of it. So this is just like collect the dataset, try it, see the loss. But then you- **Host** [1:35:56] You, you literally did this? **Eugene** [1:35:57] Yeah. At least for RWKV and the- **Host** [1:35:59] That's a lot of work **Eugene** [1:35:59] ... CodeGen models. Yeah. It's not a cheap book too, because you need GPUs. **Host** [1:36:03] Okay. And that take, that took you how long? **Eugene** [1:36:06] Oh, I took s- I think I, I, CodeGen alone was like six months, and then this RWKV, I've been doing this for like another six months. **Host** [1:36:14] Okay. **Eugene** [1:36:15] And, and that is the-- it's just pure experimentation. Like, there's no right or wrong because like, especially if it's in a different domain. **Host** [1:36:23] Yeah. **Eugene** [1:36:23] Like, like recently I was like helping someone on the RWKV Discord regarding the music, music generation domain, and my assumptions for learning rate and all the patterns, right, were just completely thrown out the window because the music model just fundamentally is different- **Host** [1:36:37] Mm-hmm **Eugene** [1:36:38] ... in, in those sense. So that is the... The exciting thing is because it doesn't really have any specific rules and guidelines- **Host** [1:36:45] Okay **Eugene** [1:36:45] ... until you get cer- until you trial and error to a certain space. It also means that you coming in is as fresh as anyone else coming in last year. It's really that kind of uncharted space for, uh, for everyone. And, and especially as you start exploring to new domains, your existing, your existing knowledge may actually matter. Because sometimes, like, I mean, I think a few papers already covered this, that, uh, like how you train your model in certain sequences also matter. Like you wanna train a certain set of knowledge, and then, and then you extend that knowledge subsequently. But if you're talking about material science or genetics, how am I supposed to know what is foundational knowledge and what is extended knowledge? I have no idea. Maybe you do. Um, I'm just picking an example. Uh, uh, yeah. So, and the same thing for music and so on. So those are things where even though you're outside the space, it's where you can come in just at the dataset level. Now you want to peel off to the next layer, let's just say. Let's just say you want to look into modifying the model, the, the, the, the, the, the foundations of it. Um, I think one of the beauties about the, this current boom is that even though I s- dip my toes early, like before the transformer wave and in the early neural network phase, frankly, almost everything that is, that matters, uh, was basically in the past four years. Like, there were a lot of things that be in academics there were before that, and in all-- and they were mostly dealing in models that were under a billion parameters. They pretty much no longer matter, and- **Host** [1:38:24] Can you be more specific? Are you talking about, like, concepts like dropout, uh- **Eugene** [1:38:29] It's c- dropout surprisingly is coming back, but like, like things like for example, like, okay, I know I'm shooting myself in the foot because R- RWKV is a neural network, but if you're just trying to get transformers to work, you don't need to know LSTM. **Host** [1:38:43] Yes. **Eugene** [1:38:43] Yeah. You don't need-- Yeah, there's a lot of like pre-knowledge in neural networks that is irrelevant in the transformer era, and maybe some of it will have a resurgence. Uh, uh, but to get up and running is not a requirement. And, and I think this is where you could either go the very academic way of reading papers and stuff, but frankly, what I found was way more useful was, uh, I, I can't pronounce this name again. The... **Host** [1:39:15] Karpathy. **Eugene** [1:39:16] Yeah, Karpathy, yeah. His series of videos- **Host** [1:39:18] Zero to Hero, yeah **Eugene** [1:39:19] ... yeah, that is really, really good. I think even if I r- even though I read some of the, read some of the papers and guides before that, it really helps that it starts from zero, because you can see how it happens part by part. And, and even though we will not use how the exact same code that he used because, like, he re-implemented the backprop and all that, and we are just gonna use Torch for that. **Host** [1:39:45] Yeah. **Eugene** [1:39:46] Uh, Py- uh, PyTorch for that. Um, that, that's where you get the aha moments on how these building blocks work and how it fall into place. And like, like, I had fundamental misunderstanding of how backprop worked until I actually watched this video. **Host** [1:40:02] Oh, really? Okay. **Eugene** [1:40:03] Yeah. And, and, and, and I think that's the scariest and craziest thing about ML sometimes is that you can actually have fundamental misunderstandings, but as long as you make the building blocks and you connect them, okay, loss is great. It works. **Host** [1:40:18] I, uh, yeah. Well, so, you know, even the gods of the industry, you know... I don't know if you read the SwigLu paper. Uh, so there's these like, these al- alternative activation functions. **Eugene** [1:40:28] Mm-hmm. **Host** [1:40:28] Like, there's ReLU, uh, and then there are people are al- always are looking for different slopes. Um, and very famously, the SwigLu paper had this line in there that was like, "Yeah, we don't know why this works, but it works." Can't explain it. **Eugene** [1:40:43] Yeah, it literally happens, uh, here and there in RWKV as well, too. One, one of the funny things that we're, that I'm doing right now in RWKV5 experiments is that, okay, we are gonna do this change where we're gonna run this train. Make your prediction. Will this model beat this model in this loss curve? **Host** [1:41:01] As a game? As a betting- **Eugene** [1:41:03] It, it's a very informal- **Host** [1:41:04] Yeah **Eugene** [1:41:05] ... it's, it's, it's, it's literally a buddy kind of like- **Host** [1:41:08] Right, right **Eugene** [1:41:08] ... kind of bet. **Host** [1:41:09] Yeah, yeah. **Eugene** [1:41:10] But, but it, it, it-- The fact that, the fact that we can do these kind of bets even though we, like, understand the code, it's like- **Host** [1:41:18] Mm-hmm **Eugene** [1:41:18] ... just goes to show, like, how often, like, oh, wait, this didn't go the way we predicted. **Host** [1:41:22] Yeah. **Eugene** [1:41:22] No one... And, and, and that's why, and that's why even if, let's say, you don't have a PhD or so on and so forth, right? Heck, even if y- math is not your specialization, you're coming in as a developer. I'm gonna come in, I'm gonna say frankly that I, I didn't come from the research background. The extremely math-heavy stuff is what I struggle with. **Host** [1:41:41] Mm. **Eugene** [1:41:41] Uh, my s- what I do sometimes is I copy and paste the math into GPT-4 and ask it to explain to me. **Host** [1:41:48] Which is good. **Eugene** [1:41:49] In, in plain old language. **Host** [1:41:50] It's very good at that. **Eugene** [1:41:51] Yeah. **Host** [1:41:51] So that's fine. **Eugene** [1:41:52] Yeah. And so but the thing is, there is lots of value add beyond that. Uh, one thing that I realized, and this is not specific to RWKV, uh, this, this also hap- happens across a lot of open source models, is that a lot of ML scientists, when they, when they really build this stuff, the focus was more of like always get it to work. It was never about getting it, it to work efficiently or getting the code documented or organized. **Host** [1:42:19] Mm. **Eugene** [1:42:20] And Stable Diffusion literally went through this whole journey. **Host** [1:42:22] Mm. **Eugene** [1:42:22] They had the giant-- They had the code and the model that worked. **Host** [1:42:26] Mm-hmm. **Eugene** [1:42:26] And the community just started, and, and engineers that came in with zero machine learning background, uh, started picking it apart and said, "No, this, this, you sh- could replace this with this that does the s- exact same thing but is more efficient." Like, one of the major breakthroughs, for example, for, for, uh, GGML, uh, recent... and this happened, happened some time back for a bit, Llama models and that, was that one, someone external from the AI community went in and implemented memory mapping. **Host** [1:42:56] Yes, I saw that. Um, yeah. I forget her name, but yeah. **Eugene** [1:42:59] Yes. **Host** [1:43:00] Just-justine.law is her URL. **Eugene** [1:43:03] Yeah. And her-- She didn't come in as an AI expert. She came in as a software engineer. **Host** [1:43:08] Yeah. **Eugene** [1:43:09] And- **Host** [1:43:09] These are all just very, very straightforward. Y- you know, in her world, this is fam- this is normal. Whereas for the researchers, they will be like, "Yeah, I don't know what that..." **Eugene** [1:43:17] Wait, what is memory map? **Host** [1:43:18] Yeah, exactly. **Eugene** [1:43:19] Yeah. And there are a lot of things like, like one of the jokes that I have right now is that, um, every month there is a research, uh, ML scientist that is rediscovering the number 32. **Host** [1:43:31] Why? **Eugene** [1:43:31] Because, uh, the-- be it like, or someone in the community writing the inference code. Because GPUs, uh, especially, uh, especially- **Host** [1:43:39] Nvidia **Eugene** [1:43:40] ... CUDA, Nvidia GPUs tends to work really well when they align to the batch size of multiples of 32. No. And if you've been in the gaming industry, especially when you write shader code, right, this is like, like well-known, like just- ... just given knowledge. And, and, and, and people are just constantly rediscovering, "Oh, maybe if I just adjust my dataset or my data size to, to fit this batch size, suddenly I get 10% improvement." And, and yeah. And then it's like these are things that once again, because they were so focused on just making it work, that they wouldn't know outside space. And that's why I would say, right, if anything right now is the best time, if that you don't know AI, to have people from different background come in because y- your contribution could be from dataset level, help train the knowledge, to shader code, to heck, um, how to memory map, how to cache data. **Host** [1:44:37] Mm-hmm. **Eugene** [1:44:37] There's so many gaps. **Host** [1:44:39] Uh, building the UI, you know, I saw that you guys have a UI as well. Uh- ... or maybe it's not maintained. I don't know. **Eugene** [1:44:44] No. Yeah. There, there's someone in the community, yeah. **Host** [1:44:46] Yeah. Um, cool. **Eugene** [1:44:48] Uh, so many areas. **Host** [1:44:48] I think that's, yeah, it's very encouraging and good to know. Uh, and then I think the last thing, you know, I, I'm ... Th- I left this to the end because it's kind of uncomfortable, but also just fun bonus, which is the, uh, I, I'm really trying to do an AI Waifu episode. ### Mind Upload **Host** [1:45:04] Uh, I think that, at least in the open source model space, the most motivated and surprisingly competent, um, people are the people trying to build AI girlfriend. Um, and you are one of the few people I've ac- actually met who, like, you interact with these people, right? Like, that then they are ... Like, just what are you seeing? What, what's interesting? Like, and there's ... Apart from R- RWKV, there's also other communities, right? **Eugene** [1:45:27] Right. **Host** [1:45:27] The, the uncensored models. Um, I think WizardLM is part of that. **Eugene** [1:45:31] Correct. **Host** [1:45:31] Um, just like can you, can you sketch out like what is happening in that world? **Eugene** [1:45:37] So, uh, I mean, credit where credit is due, right? We, uh ... Like this is, there's ... We shouldn't be king shaming or anything on that, and these are some of the most motivated and even sometimes even the most technical competent people- **Host** [1:45:52] Yeah **Eugene** [1:45:52] ... that literally move mountains in the code base. Uh, and I don't mean that lightly. Uh, it's like, it's like, um, I, I, I think those active in the RWKV, uh, Discord will know, will know a few members that literally just came in out of nowhere. And, and it's like, "Okay, let's just rewrite the whole how CPP and GGML code does work," and- ... and great, it's way faster. A- a- and, and, um, uh, there's, there's a lot of ... The, a lot of them, their motivations is still very inherently is that they are very ... I guess it's the fastest feedback loop from code. **Host** [1:46:32] They are the users. Yeah. **Eugene** [1:46:33] To the user. Yes. **Host** [1:46:34] Yeah, yeah. **Eugene** [1:46:34] Exactly. And, and, and they want, they want the model to align better. So the ... And the thing is getting an AI Waifu actually spreads the whole freaking domain. **Host** [1:46:45] Why? **Eugene** [1:46:46] Because, um, from the fun- uh, from the very top, from the very bottom right would be, let's say the, the model architecture. So let's say if the model architecture has issues paying attention to historical conversations, for example, um, you, you, you can have long conversations and then the model will just forget stuff. **Host** [1:47:04] Yes. **Eugene** [1:47:05] Not, not, not ideal, per se. Uh, all the way to the very top would be like, like you want your model to stay in character, your system prompts. Th- this is literally alignment problems, but the alignment is not to an ethical standard. Alignment is to stay in character. And that includes doing things that makes no sense. Like let's say, let's just say we take your, one of your favorite, uh, one, one of your favorite, um ... Like what's the character trope for this? Uh, uh, the silly, silly scientist or silly e- e- airhead, airhead girl. Uh, I think the American equivalent would be Dumblo- **Host** [1:47:43] Bimbo **Eugene** [1:47:43] ... Dumbblonde. **Host** [1:47:44] Yeah, bimbo. Sorry. **Eugene** [1:47:45] Yeah. N- I'm sorry if I offend- **Host** [1:47:46] Yeah, yeah **Eugene** [1:47:47] ... person. And the idea that you ... The idea there is that the characters may make, as in, in character, will make some very silly mistakes. **Host** [1:48:00] Yeah. **Eugene** [1:48:00] And you want to align your model that way. So that's alignment. **Host** [1:48:04] So, okay, what are people doing to solve that? **Eugene** [1:48:06] Um- **Host** [1:48:07] Just in case you've seen anything interesting. **Eugene** [1:48:08] So- **Host** [1:48:09] Like for example, the Dan prompt to me was very interest- was very interesting. **Eugene** [1:48:12] Yeah. **Host** [1:48:12] Like, like give people points and then deduct points, and like it's trained to be very scared of losing points. **Eugene** [1:48:17] Correct. Uh, so, so from there onwards, it's really more of like prompt chaining methods. They are, uh, embedding- **Host** [1:48:25] Which makes it slower **Eugene** [1:48:26] ... which makes it slower. And then so, so it keeps going back and forth the chain. So you see, they, they make the ... They adjust the prompt, then it's too slow, then they want to optimize it, then they look, look into how to train it better datasets and including their favorite character stories- ... uh, from whatever sources they can get. Because one of the existing problem for, for AI models, even from the foundation model, right, is that even though it can partially impersonate a character- **Host** [1:48:51] Uh-huh **Eugene** [1:48:53] ... uh, if you ask a real fan, in a lot of cases it falls flat. Because what's happening is it's reading summaries and quotes and memes- **Host** [1:49:01] Yeah **Eugene** [1:49:02] ... and impersonating at a very high level. **Host** [1:49:05] Yeah. **Eugene** [1:49:05] But it's not impersonating on a very deep level. **Host** [1:49:08] Yeah. **Eugene** [1:49:08] And, and that's where, where, where people start exploring the dataset. And, and because these commit- these members are also the same members that do not have a giant GPU farm, they are very interested in optimizing it- **Host** [1:49:20] Prompts **Eugene** [1:49:20] ... be it through LoRA- **Host** [1:49:21] Okay **Eugene** [1:49:22] ... or fine-tuning and it's like what's the best learning rate? What's the best way to fine-tune this limited GPU resource- **Host** [1:49:28] Yeah **Eugene** [1:49:29] ... for the benefit of all, all people. **Host** [1:49:31] Are the LoRA techniques and whatever else, are, are they applicable to RWKV? **Eugene** [1:49:35] Yeah. Uh, RWKV does have a Lo- a LoRA trainer as well. **Host** [1:49:38] Okay. **Eugene** [1:49:39] And- **Host** [1:49:40] And that's relatively commonplace now. Everyone has it. **Eugene** [1:49:43] Yeah. I think pretty much every open source model has a LoRA trainer. 'Cause the m- **Host** [1:49:46] I, I, I will say I, I've actually struggled to find ... Like LoRA seems to be very common in the Stable Diffusion community, but in, in text models, I haven't really seen that much adoption in, in my circles. But I, I think maybe you've seen more. **Eugene** [1:49:59] I guess the problem is that LoRA has- Okay, so like, I think Stable Diffusion LoRA is a lot more powerful, as in I'm try- I find it hard to come up with a use case that LoRA cannot support. But for example, right, for example, in, in, in, in the language model's case, right, LoRA cannot teach new language. Uh, it cannot, uh... It sometimes may struggle to, to teach new techniques or new, new concepts. Uh, it does well into editing and refining existing context, uh, knowledge. And that-- And the part where-- Then this is the part where how do we know whether it works or doesn't? We don't really know, because the line is very gray, and I think that frustrates a lot of people- **Host** [1:50:46] Mm-hmm **Eugene** [1:50:47] ... when they're using LoRA for professional use. Because you can end up doing LoRA and foil the 4th letter completely. **Host** [1:50:54] Mm. Mm. **Eugene** [1:50:55] But this is where back to the character AI community. **Host** [1:50:59] Mm-hmm. **Eugene** [1:51:00] It's actually very suited for, for that use case, because if your character is popular enough, there is some base data in there. **Host** [1:51:06] Yeah. **Eugene** [1:51:06] And you're just effectively fine-tuning the speech patterns and the data from there. **Host** [1:51:11] Yeah. So I'll, I'll call out, um... So I think you say character AI, but we-- you don't actually mean the company Character AI, right? **Eugene** [1:51:17] Oh, yeah. Yeah. Sorry about that. **Host** [1:51:18] It's the, it's the, the companies that are like them, but, uh, sex positive, let's, let's just say. **Eugene** [1:51:23] Okay. Yeah. **Host** [1:51:24] Whatever. So there's Character AI, there's Replica. These are the two, like, trad... I would con- consider them, like, trad in terms of, like, they are in the common consciousness in, at least in, at least in traditional AI circles. **Eugene** [1:51:35] Yeah. **Host** [1:51:35] And then, uh, for example, I recently came across venus.chub, which, yes, it's one of those. But, like, two million users in one week. Uh, that- that's the number that I got. Um, crazy. Like, just huge. **Eugene** [1:51:50] Yeah. And there's a-- I think there's also a lot of it, especially when it comes to, like, specific domains. **Host** [1:51:55] Yeah. **Eugene** [1:51:56] Like be it anime, be it- **Host** [1:51:58] Furries. **Eugene** [1:52:00] There- **Host** [1:52:01] These are, these are all, like... Look, like, I mean, this, this is all the, the full range. You wanna simulate humanity, yeah, there you go. **Eugene** [1:52:08] Fair enough. **Host** [1:52:09] A lot of the, a lot of times it's about sex. **Eugene** [1:52:10] Yeah. **Host** [1:52:11] Um, okay. So, uh, any, uh... I don't know if you have anything else. I- I'll, I'll mention, like, one other piece of why I'm interested in this is because these people, um, could be... Actually, honestly, they are the pioneers in terms of modeling what a human is, and you might actually-- we actually end up figuring out how to, like, encode a human personality and, and, and, uh, identity. Um, and we might actually end up, like, this weird path that we're taking might actually end up in, like, mind uploading, which is what I'm thinking about. **Eugene** [1:52:44] I don't think... Yeah, I think that makes sense in, in many ways because they are also the most nitpicky about it. **Host** [1:52:50] Yes. **Eugene** [1:52:51] It's like they can tell when a character is out of character. **Host** [1:52:56] Yeah. So, so, you know, a- a- and they're doing it without access to the full information. But, like, I, I do think that, like, this is a real path towards, like, immortality in, in some form. Um, and I think there will be people interested in mind upload, and it will come from this community because no one else is working as, as hard on, uh, essentially serialization of a person. **Eugene** [1:53:16] I think there's, there are two variants for it. Like, uh, I think one is the one that Facebook, uh, is attempting, uh- **Host** [1:53:23] Okay **Eugene** [1:53:23] ... which is I have all the data on you. **Host** [1:53:26] Yeah, yeah. **Eugene** [1:53:26] And when, when, and same thing, I have all data on this character. **Host** [1:53:30] Yeah. **Eugene** [1:53:31] And now I, now you have a virtual half, per se. And your, your, your... When you decease, whatever's left can interact with that. Uh, that, I think that's slightly different from mind upload. But then subsequently, I think then the next jump would be... But that could be, like, the building block to the next major jump, which is, like, really scanning your brain and then figuring out- **Host** [1:53:54] Yes **Eugene** [1:53:54] ... how all this connects. And- **Host** [1:53:57] And sequence your DNA- **Eugene** [1:53:58] Yeah **Host** [1:53:58] ... like, do whatever, right? **Eugene** [1:53:59] Just, like, like, this is, like, a completely wild tangent, right? Like, like, uh, I sometimes think that we overestimate how far we are, uh, because in my opinion, right, like... And this is for me, in particular with the Stable Diffusion model, is that if I can get the world image model effectively Stable-- I mean, Stable Diffusion was, uh, in under 100 gigabytes, and now I have all the world knowledge literally in a transformer that's less than 100 gigabytes. No offense to myself, I don't think my personality and my memories is- **Host** [1:54:41] Yeah **Eugene** [1:54:41] ... more than this. **Host** [1:54:41] Yeah. **Eugene** [1:54:42] We co- even if I 10x it, I could store this in two SSDs. Two hard drives. **Host** [1:54:48] Yeah. **Eugene** [1:54:50] And, and if we really break it down how to serialize it and handle it right, perhaps we are actually not as big as we think we are. **Host** [1:54:59] Yeah, yeah. **Eugene** [1:54:59] Because our brains are actually handling a crap ton of other functions, and they, and, and this is, like, a tangent to the biological side. Like, like, your, your whole body, your breathing- **Host** [1:55:09] Your breathing- **Eugene** [1:55:09] Yeah **Host** [1:55:09] ... you're pumping blood. **Eugene** [1:55:10] Your movement. **Host** [1:55:11] Yeah. **Eugene** [1:55:11] That actually takes up a lot. **Host** [1:55:12] Yeah. **Eugene** [1:55:13] And if you really want to strip it down to, like, just pure text and vision, because since now if you upload your mind, you no longer need the rest of that. **Host** [1:55:21] Yeah. **Eugene** [1:55:22] Perhaps we may find out that it's actually a lot less than we think. **Host** [1:55:25] Yeah. So, uh, George Hart was on our podcast. **Eugene** [1:55:27] Oh. **Host** [1:55:27] And, and he said two gigs. **Eugene** [1:55:29] Two gigs. **Host** [1:55:30] He's, he wants to quantize himself, which I'm like, "I think you'll lose something if you quantize yourself." But- **Eugene** [1:55:36] I, I wouldn't push so far to do it. I'm, I'm still within this terabyte range because frankly, that's all we need, right? **Host** [1:55:41] That's all we need. That's all we need. Yeah. Yeah. Um, cool. Great. Um, so yeah, thanks so much for being, uh, very, uh, willing to get on and talk with no prep. Uh- **Eugene** [1:55:49] Yeah **Host** [1:55:49] ... we, we did some prep, but, uh, it's very unusual podcast episode, but I really enjoyed it, and hopefully- **Eugene** [1:55:54] We literally just met yesterday- **Host** [1:55:56] Yeah **Eugene** [1:55:56] ... in Singapore. **Host** [1:55:56] Well, but I know you've been on the Discord for a while, and- **Eugene** [1:55:58] Yeah **Host** [1:55:58] ... I, I, I could tell you, like, you're very serious about, uh, all this. Uh, I think it's very unusual for so- for someone... Like, you have a job, but, but this is, like, a second job essentially. **Eugene** [1:56:07] Yes. **Host** [1:56:08] You know? Um, but you are really, um, enthusiastic and passionate about it, and I think that's very rare, and I, I'd wanna encourage more people to, to do it. And, uh, so thanks for sharing. **Eugene** [1:56:17] Yeah. Thank for having me here on, uh, very last minute basis. Like, we did not book this room. We are, we are- **Host** [1:56:23] There's no room. **Eugene** [1:56:24] We are literally, we are literally gorilla podcasting in some corner. **Host** [1:56:28] Yeah. **Eugene** [1:56:28] So if you see random intermissions and cut, right- **Host** [1:56:30] Yeah **Eugene** [1:56:30] ... that was because a crowd just went by and there was a noise, and we needed to pause. **Host** [1:56:34] Aunties had to go for lunch. Uh, but no, I, I think it's actually a bit charming. You know, I think some podcasts can be too polished, and sometimes it's just nice to see, like, hey, it's just two guys who's talking. **Eugene** [1:56:44] Yeah. **Host** [1:56:44] Yeah. That's all it is. Uh, cool. Thanks. **Eugene** [1:56:47] All right. Thanks for having me here. All right. Oh, you s- --- This library is powered by PodHood (https://podhood.com), the podcast website platform.