# Personal benchmarks vs HumanEval - with Nicholas Carlini of DeepMind Latent Space · 2024-08-28 Nicholas Carlini, a DeepMind research scientist, argues AI models are practically useful for personal tasks despite flaws, and personal benchmarks tailored to individual use cases matter more than generic leaderboards. He shares from his 'How I Use AI' post: using LLMs for ephemeral software, kickstarting Docker, debugging by pasting error messages. He built a personal benchmark DSL that runs code from real chat history. On security, he explains buying expired domains from LAION-400M allowed poisoning any model trained on it, and he extracted OpenAI's Ada and Babbage model dimensions via API (with permission). He also recovered training data from GPT-3.5 by repeating a word until ChatGPT output verbatim sequences. He prefers attacking over defending because it's more fun and essential for discovering real vulnerabilities. ## Questions this episode answers ### Why does Nicholas Carlini recommend creating personal LLM benchmarks instead of relying on standard ones like HumanEval? Carlini argues standard benchmarks measure tasks no one actually cares about, while personal benchmarks are built from your own chat history, testing models on problems you genuinely face. He created a domain-specific language to automate this, letting him quickly evaluate new models on tasks relevant to him and bypass benchmark overfitting. [37:42](https://addtry.com/fe664320-4804-4d5e-bf84-004aee23af53?t=2262000) ### How did Nicholas Carlini steal a layer from OpenAI’s models, and what was the impact? Carlini and his co-authors exploited the logprobs output with a logic bias to extract the projection matrix of one layer from models like GPT-3.5 Turbo, with OpenAI’s permission. The vulnerability led OpenAI to remove certain API features to prevent such model theft, marking a rare case where real-world ML systems were hardened in response to adversarial attacks. [55:46](https://addtry.com/fe664320-4804-4d5e-bf84-004aee23af53?t=3346000) ### What does Nicholas Carlini mean by 'ephemeral software' in the context of LLMs? Carlini uses LLMs to generate throwaway scripts for one-off tasks—like data conversion or quick visualizations—that would not be worth writing manually. This lowers the effort from 45 minutes to 5, enabling him to explore more ideas. If the model fails, he simply moves on, but when it succeeds, it dramatically speeds up his workflow. [14:09](https://addtry.com/fe664320-4804-4d5e-bf84-004aee23af53?t=849000) ## Key moments - **[0:00] Intro** - [1:46] Nicholas Carlini built a Doom clone in JavaScript as a fun side project. - [2:28] Nicholas Carlini built multiplexing circuits in Conway's Game of Life to demonstrate Turing completeness. - [3:56] Calling printf in C is Turing complete, as Nicholas Carlini demonstrated by overwriting its own format indexing with %n. - **[5:03] How I Use AI** - [5:39] Nicholas Carlini wrote 'How I Use AI' to counter both AI hype and AI skepticism with concrete personal use cases. - [7:43] "I probably spent, I don't know, definitely less than 10 hours putting this together." - [8:47] Q: Where are we in the spectrum between ChatGPT being enough to build an app versus needing a full AI agent? - [11:15] Nicholas Carlini used AI to learn Docker on demand, avoiding deep dives into tools he only needed temporarily. - [14:04] LLMs lower the activation energy for creating ephemeral, single-use software, making 5-minute throwaway programs worthwhile. - [15:35] "When you're using these models, if you're getting the answer you want always, it means you're not asking them hard enough questions." - **[30:12] AGI Speculation** - [30:12] Carlini plans a post arguing that both 'AGI by 2027' and 'LLMs are the wrong path' are extreme; a nuanced middle exists. - [32:50] Carlini avoids social media on principle and distributes his writing solely via RSS and an email list. - [34:52] Carlini plans to post unrelated content to his blog to shed subscribers who only want AI takes. - **[35:39] Paper Data** - [35:39] Carlini's project encodes 1.44 megabytes on a single printed page, matching a floppy disk's capacity. - **[37:37] Personal Benchmarks** - [37:42] Carlini built a personal AI benchmark because public benchmarks don't measure what he personally cares about. - [38:27] Carlini notes that ImageNet's 300 dog breed classification was never practically useful but drove progress. - [39:10] Carlini advocates for personal benchmarks to avoid model over-optimization on popular tests. - [43:04] Q: How does Nicholas Carlini see the evolution of prompt engineering? - [45:24] Two years ago you had to prompt 'think step by step'; now models self-prompt, says Carlini. - [48:26] Carlini agrees single-turn evals miss real chat interactions; he's exploring multi-turn benchmarks. - **[52:14] Security Attacks** - [52:14] Carlini bought expired domains used by the LAION-400M dataset to demonstrate how easy it is to poison training data. - [52:53] Carlini criticized adversarial ML for studying unrealistic attacks; he focuses on practical exploits against real systems. - [55:33] With permission from OpenAI and Google lawyers, Carlini stole a layer of OpenAI's models by exploiting logit bias APIs. - [58:02] Carlini's team disclosed the model stealing vulnerability to OpenAI and others, giving 90 days to fix before publishing. - [59:11] OpenAI removed useful APIs to prevent Carlini's model stealing attack, a rare case of security winning over utility in ML. - [1:01:29] Q: Can LLMs be prompted to reveal the proprietary data they were trained on? - **[1:03:30] Motivation** - [1:03:30] Q: Why does Nicholas Carlini prefer attacking AI systems over building defenses? - [1:06:09] Carlini's PhD required a defense chapter for graduation; he found it unmotivating and never published it separately. ## Speakers - **Alessio** (host) - **Swyx** (host) - **Nicholas Carlini** (guest) ## Topics Benchmarks, Security ## Mentioned DeepMind (company), OpenAI (company), ChatGPT (product), Claude (product), GPT-4 (product), Gemini (product), LAION-400M (product) ## Transcript ### Intro **Alessio** [0:04] Hey, everyone. Welcome to the Latency-Based Podcast. This is Alessio, partner and CTO in residence at Decibel Partners, and I'm joined by my co-host, Swyx, founder of Small AI. **Swyx** [0:13] Hey, and today we're in the in-person studio which, uh, we- Alessio has gorgeously set up for us, uh, with Nicholas Carlini. Welcome. **Nicholas Carlini** [0:21] Thank you. **Swyx** [0:22] Uh, you're a research scientist at DeepMind. You work at the intersection of machine learning and computer security. You got your PhD from Berkeley in 2018, and also your BA in, uh, from Berkeley as well. And mostly we're here to talk about your blogs because you are so generous in, in just writing up what you know. Well, actually, why do you write? **Nicholas Carlini** [0:41] Because I like... I feel like it's fun to share what you've done. I don't like writing. Sufficiently didn't like writing, I almost didn't do a PhD because I knew how much writing was involved in writing papers. **Swyx** [0:51] Uh-huh. **Nicholas Carlini** [0:52] I was terrible at writing when I was younger. I do like the remedial writing classes when I was in university 'cause I, I was really bad at it. Uh, so I don't actually enjoy, I still don't enjoy the act of writing, but I feel like it is useful to share what you're doing, and I like being able to talk about the things that I'm doing that I think are fun. And so I write because I think I want to have something to say, not because I enjoy the act of writing. But yeah. **Swyx** [1:15] It's a tool for thought as they, as they often say. Is there any sort of backgrounds or thing that people should know about you as a, as a person? Like, you know, just... **Nicholas Carlini** [1:23] Yeah. So I, I tend to focus on, like you said, I do security work. Uh, I try to like attacking things and I want to do like high quality security research and that's mostly what I spend my actual time- **Swyx** [1:37] Yeah **Nicholas Carlini** [1:37] ... trying to be a productive member of society doing that. But then I get distracted by things- ... and I just like, you know, working on random fun projects and so- **Swyx** [1:44] Like a Doom clone in JavaScript. **Nicholas Carlini** [1:46] Yes, like, like that. Or, um, you know, I, I've done a number of, yeah, very sort of things that have absolutely no utility but are, are fun things to have done. And so it's interesting to say like you should work on fun things that just are, are interesting even if they're not useful in any real way and so that, that's what I tend to, to put up there is after I have completed something I think is fun or if I think it's sufficiently interesting, write something down, down, down there. **Alessio** [2:10] Before we go into like AI, LLMs and whatnot, why are you obsessed with the Game of Life? So you, you built multiplexing circuits in the Game of Life which is mind-boggling. So where did that come from and then how do you go from just clicking boxes on the UI web version to like building- **Nicholas Carlini** [2:28] Sure **Alessio** [2:28] ... multiplexing circuits? **Nicholas Carlini** [2:30] I like Turing completeness. The definition of Turing completeness is a computer that can run anything essentially, and the Game of Life, Conway's Game of Life is a very simple cellular 2D automata where you have cells that are either on or off and a cell becomes on if in the previous generation some configuration holds true and off otherwise. And it turns out there is a proof that the Game of Life is Turing complete, that you can run any program in principle using Conway's Game of Life. I don't know and so you, you can therefore someone should and so I, I- ... wanted to do it and, um, you know some other people have done some similar things but I got obsessed into like... If you're gonna try and make it work, like we already know it's possible in theory. I want to try and like actually make something I can run on my computer like a real, a real computer I can run and so yeah I, I, I've been going down this rabbit hole of trying to make a, a CPU that I can run semi-real time on the Game of Life and I have been making some reasonable progress there and yeah. Um, but you know Turing completeness is just like a very fun trap you can go down. Um, a while ago as part of a research paper I was able to show that in C if you call into printf it's Turing complete. Like printf you know like which like you know you can print numbers- **Alessio** [3:39] Yeah, yeah, yeah **Nicholas Carlini** [3:39] ... or whatever. Right. Yeah. **Swyx** [3:40] Yeah. But there shouldn't be no like control flow stuff in there. **Nicholas Carlini** [3:42] There is because printf has a %n specifier that lets you write an arbitrary amount of data to an arbitrary location and the printf format specifier has an index into where it is in the loop that is in memory. So you can overwrite the- **Alessio** [3:58] Mm **Nicholas Carlini** [3:58] ... in, the location of where printf is pr- currently indexing using %n. So you can get loops, you can get conditionals- ... and you can get arbitrary data rates again. So we sort of have another Turing complete language using printf. Which again like, like this has um, essentially zero practical utility- **Alessio** [4:13] Yeah, yeah, yeah **Nicholas Carlini** [4:13] ... but like it's just, I feel like a lot of people get into programming because they enjoy the art of doing these things and then they go work on developing some software application and lose all joy. **Swyx** [4:25] Need a new SaaS with the boys- **Nicholas Carlini** [4:27] Yeah **Swyx** [4:27] ... as they say. **Nicholas Carlini** [4:27] And I wanna still have joy in doing these things and so on occasion I try to stop doing productive meaningful things and just like what's a fun thing that we can do and try and make that happen. **Alessio** [4:40] Awesome. And you've been kinda like a pioneer in the AI security space. You've done a lot of talks starting back in 2018. We'll kinda leave that to the end- **Nicholas Carlini** [4:48] Sure **Alessio** [4:49] ... um, because I know the, the security part is uh, there's maybe a smaller audience but it's a very intense audience so I think that'll be fun. But everybody in our Discord started posting your how I use AI blog post and we were like, "We should get Carlini on the podcast." And uh, and then uh- ### How I Use AI **Swyx** [5:04] And you were so nice to just- **Alessio** [5:05] Yeah and then I sent you an email- **Nicholas Carlini** [5:06] I was like, "What?" **Alessio** [5:06] ... and you're like, "Okay, I'll come." And I was like, "Oh, I thought that would be harder." **Swyx** [5:09] That was like that was easy. **Alessio** [5:10] So I think there's as you said in the blog post a lot of misunderstanding about what LLMs uh, can actually be used for, uh, what are they useful at, what are they not good at, and whether or not it's even worth arguing what they're not good at because they're obviously not. So if you can knock on the Rs in a world, in a word they're like it just not what it does. So how painful was it to write such a long post given that you just said that you don't like to write? **Nicholas Carlini** [5:33] Yeah. **Alessio** [5:34] Uh, and then we can kinda run through the, the, the things but maybe just talk about the motivation why you thought it was important to do it. **Nicholas Carlini** [5:39] Yeah. So I wanted to do this because I feel like most people who write about language models being good or bad, some underlying message of like, you know, they have their camp and their camp is like AI is bad or AI is good or whatever and they like they spin whatever they're gonna say according to their ideology and they don't actually just look at what is True in the world. So I've read a lot of things where people say how amazing they are and how all programmers are gonna be obsolete by 2024. And I've read a lot of things of people who say, like, they can't do anything useful at all, and, you know, like, they're... It's just like it's only the people who've come off of, you know, blockchain crypto stuff and have... are here to, like, make another quick buck and move on. And I don't really agree with either of these, and I'm not someone who cares really one way or the other how these things go. And so I wanted to write something that just says, like, "Look, like, let's sort of ground reality in what we can actually do with these things." Because my actual research is in, like, security and showing that these models have lots of problems, like, this is, like, my day-to-day job is saying, like, we probably shouldn't be using these in lots of cases. I thought I could have a little bit of credibility of, in saying, "It is true, they have lots of problems. We maybe shouldn't be deploying them in lots of situations, and still, they are also useful." And that is the, like, the bit that I wanted to get across, is to say, "I'm not here to try and sell you on anything. I just think that they're useful for the kinds of work that I do." And hopefully, some people would listen, and it turned out that a lot more people liked it than I, I thought. Uh, but yeah, that, that was the motivation behind why I, I wanted to write this. **Alessio** [7:16] So you had about a dozen sections of, like, how you actually use AI. Um, maybe we can just kind of run through them all, and then maybe the ones where you have extra commentary to add, we can- **Nicholas Carlini** [7:27] Sure **Alessio** [7:27] ... double on. **Nicholas Carlini** [7:27] Yeah, yeah. No, this, um, I didn't put as much thought into this as, um, maybe was deserved because, you know, look, I, I probably spent, I don't know, definitely less than 10 hours putting this together. **Alessio** [7:39] Wow. It took me close to that to do a podcast episode. So that's, uh- **Nicholas Carlini** [7:42] Yeah **Alessio** [7:42] ... that's pretty impressive. **Nicholas Carlini** [7:43] Yeah. I, I wrote it in one pass. I still have... Like, I've gotten a number of emails of, like, "You got this editing thing wrong. You got this sort of other thing wrong." It's like I, I haven't, just haven't looked at it anymore. **Alessio** [7:51] Yeah, yeah. **Nicholas Carlini** [7:52] Like, I, I tend to... Yeah, try... I feel like bec- I, I still don't like writing, and so because of this, the way I tend to treat this is, like, I will put it together into the best format that I can at a time, and then put it on the internet, and then never change it. And I guess this is- **Alessio** [8:04] Mm-hmm **Nicholas Carlini** [8:04] ... the sort of aspect of, like, the research side of me, is, like- **Alessio** [8:06] Mm-hmm **Nicholas Carlini** [8:06] ... once a paper is published, like, it is done. It is an artifact. **Alessio** [8:09] Yeah. **Nicholas Carlini** [8:09] It exists in the world. I could forever edit the very first thing I ever put to make it the most perfect version of it, of what it is, and I would do nothing else. **Alessio** [8:16] Right. **Nicholas Carlini** [8:16] And so I feel like if I find it useful to be like, "This is the artifact. I will spend some certain amount of hours on it, which is what I think it is worth, and then I will just-" **Alessio** [8:22] Yeah. Time boxing. **Nicholas Carlini** [8:23] Yeah. Stop. **Alessio** [8:24] Yeah. So the first one was to make applications. We just recorded an episode with the founder of Cosine, which is like a AI software engineer colleague. You said it took you 30,000 words to get, uh, GPT-4 to build you the, "Can GPT-4 solve this?" Kind of like, uh, app. Where are we in the spectrum where ChatGPT is all you need to actually build something versus I need a full-on agent that does everything for me? **Nicholas Carlini** [8:47] Yeah, okay, so this was in, um... So I, I built a web app, uh, last year sometime that was just, like, a fun demo. You, where you can guess if you can predict whether or not GPT-4 at the time could solve a given task. This is, as far as web apps go, very straightforward. Like, you need basic HTML, CSS. **Alessio** [9:04] Mm-hmm. **Nicholas Carlini** [9:05] You put, have a little slider that moves. You have a button, sort of animate the text coming to the screen. The reason people are going here is not because they want to see my wonderful HTML. **Alessio** [9:14] Mm-hmm. **Nicholas Carlini** [9:15] Right? Like, you know, I, I used to know how to do, like, modern HTML, like, in 2007, 2008. Like, you know, and I, I was very good at fighting with IE6 and these kinds of things. **Alessio** [9:25] Mm-hmm. Yeah, yeah. **Nicholas Carlini** [9:26] Like, I knew how to do that. I have no longer had to build any web app stuff, like, in the meantime, which means that, like, I know how everything works, but I don't know any of the new- **Alessio** [9:34] Mm-hmm. **Nicholas Carlini** [9:35] Flexbox is new to me. Flexbox is like- ... 10 years old at this point. Um, but, like, it's just amazing having, being able to go to the model and just say, like, "Write me this thing," and it will give me all of the boilerplate that I need to get going. And of course, it's imperfect. It's not gonna get you the right answer, and it doesn't do anything that's complicated right now, but it gets you to the point where the only remaining work that needs to be done is the interesting hard part for me that, like, is the actual novel part. Um, and even the current models, I think, are entirely good enough at doing this kind of thing that they're very useful. It may be the case that if you had something out, like you were saying, smarter agent that could debug problems on, by itself, that might be even more useful. Currently, though, make a model into an agent by just copying and pasting error messages for the most part, and that's what I do, is, you know, you, you tr- you run it, and it gives you some code that doesn't work, and either I'll fix the code, or it will give me buggy code, and I won't know how to fix it, and I'll just copy and paste the error message and say, uh, "It tells me this. Um, what do I do?" And it will just tell me how to fix it. You can't trust these things blindly, but I feel like most people on the internet already understand that things on the internet you can't trust blindly. **Alessio** [10:43] Right. **Nicholas Carlini** [10:44] And so there's not, like, this is not, like, a big mental shift you have to go through to understand that it is possible to read something and find it useful even if it is not completely perfect in its output. It- **Alessio** [10:55] It's very human-like in that sense. It's the same ring of trust. You know, I kind of think about that, that, that way if you had, um, yeah, trust levels. And there's maybe a couple that tie together. So there was, like, to make applications, and then there's to get started, which is a similar, you know, kickstart, maybe like a project that you know the LLM cannot solve. It's kind of how you think about it. **Nicholas Carlini** [11:15] Yeah. So the, like, for getting started on things is one of the cases where I think it's, it's really great for some of these things, where I sort of use it as a personalized, "Help me use this technology I've never used before." **Alessio** [11:27] Mm-hmm. **Nicholas Carlini** [11:27] So for example, I had never used Docker before January. I know what Docker is. **Alessio** [11:31] Lucky you. **Nicholas Carlini** [11:32] Yeah, like, I, I... Like, I'm, I'm a computer security person. **Alessio** [11:34] Yeah. **Nicholas Carlini** [11:34] Like, I sort of, I, I have read lots of papers on, you know, on all the technology behind how these things work, and I know all the exploits on them. I've, I've done some of these things, but I had never actually used Docker. But I wanted it to be able so that I could run the outputs of language model stuff in some controlled, contained environment, which I know is the right application. So I just ask it, like, "I want to use Docker to do this thing. Like, tell me how to run a Python program in a Docker container." And it, like, gives me a thing, and I'm like, "Uh, step back. You said Docker Compose. I, I do not know what this word Docker Compose... Is this Docker? Is this not Docker? Help me." And, like, it'll sort of tell me all of these things, and I'm sure this knowledge is out there on the internet. Like, this is not- Some groundbreaking- **Swyx** [12:13] Mm-hmm **Nicholas Carlini** [12:13] ... thing that I'm doing, but I, I just wanted it as a small piece of one thing I was working on, and I didn't want to learn Docker from first principles. Like, I, I... At some point, if I need it, I can do that. Like, I, I have the background that I can make that happen. But what I wanted to do was, was thing one, and it's very easy to get bogged down in the details of this other thing that helps you- **Swyx** [12:34] Mm-hmm **Nicholas Carlini** [12:34] ... accomplish your end goal, and I just want, like, tell me enough about Docker so I can do this particular thing, and I can check that it's doing the safe thing. I sort of know enough about that from, you know, my, my o- my other background, and so I can just have the model help teach me exactly one thing I want to know and nothing more. I don't need to worry about other things that the writer of this thinks is important that actually isn't. Like, I can just, like, stop the conversation and say, "No, boring to me. Explain this detail I don't understand." I think that's... Well, that was very useful for me. It would have taken me, you know, several hours to figure out some things that take 10 minutes if you can just ask exactly the question you want the answer to. **Swyx** [13:06] Have you had any issues with, like, newer tools? Have you felt any meaningful kinda like a cutoff day where, like, there's not enough data on the internet or? **Nicholas Carlini** [13:14] I, I, I'm sure that the answer to this is yes, but I tend to just not use most of these things. **Swyx** [13:20] Those stuff. **Nicholas Carlini** [13:20] Like, I feel like this is... This is... Like, the significant way in which I use machine learning models is probably very different than most people, is that I'm a researcher and I get to pick what tools that I use, and most of the things that I work on are fairly small projects. And so I can, I can entirely see how someone who is in a big, giant company where they have their own proprietary legacy code base of, of 100 million lines of code or whatever and, like, you just might not be able to use things the same way that I do. I still think there are lots of use cases there that are entirely reasonable that are not the same ones that I've put down, but I wanted to talk about what I have personal experience in being able to say is useful, and I would like it very much if someone who is in one of these environments would be able to describe the ways in which they find current models useful to them and not, you know, philosophize on what someone else might be able to find useful. Actually say, like, "Here are real things that I have done that I, I, I found useful for me." **Swyx** [14:09] Yeah. Th- this is what I often do to encourage people to write more, uh, to share their experiences, because they, they often fear, uh, being attacked on the internet. But you are the ultimate authority on how you use things, and there's... it's objectively true, so that cannot be debated. One thing that people are very excited about is the concept of ephemeral software or, like, pers- like, personal software. This use case in particular basically lowers the activation energy for creating, uh, software, which I like as a vision. I don't think I, I have made advan- as much adva- uh, taken advantage, as much advantage of it as I could. I feel guilty about that, but also we're, we're trending towards there. **Nicholas Carlini** [14:48] Yeah. No, I mean, this... I, I do think this is a direction that, that is exciting to me. Uh, yeah, one of the things I wrote was, like, a lot of the ways that I use these models are for proj- like, for one-off things that I just need to happen that I'm gonna throw away in five minutes. **Swyx** [15:00] Yeah. **Nicholas Carlini** [15:01] And- **Swyx** [15:01] And you can. **Nicholas Carlini** [15:02] Yeah, exactly. Right. It's like the kind of thing where it would not have been worth it for me to have spent 45 minutes writing this because I don't need the answer that badly. But if it will only take me five minutes, then I'll just figure it out, ru- run the program, and then see and then get it right. And if, and if it turns out that you ask the thing, it doesn't give you the right answer, well, I didn't actually need the answer that badly in the first place. Like, either I can decide to dedicate- **Swyx** [15:23] Mm-hmm **Nicholas Carlini** [15:23] ... the 45 minutes or I cannot, but, like, the cost of doing it is fairly low. You, you, you see what the model can do, and if it can't, then okay. When you're using these models, if you're getting the answer you want always, it means you're not asking them hard enough questions. And- **Swyx** [15:36] Ooh. Say more **Nicholas Carlini** [15:37] ... lots of people only use them for very small particular use cases, and the... like, it always does the thing that they want. **Swyx** [15:43] Yeah, like a... They use it like a search engine or, you know. **Nicholas Carlini** [15:44] Yeah. Well, like one particular case. **Swyx** [15:46] Yeah. **Nicholas Carlini** [15:46] And, and if you're finding that when you're using these it's always giving you the answer that you want, then probably it's has more capabilities than you're actually using. And so I oftentimes try, when I have something that I'm curious about, to just feed into the model and be like, "Well, maybe it'll solve my problem for me." You know, most of the time it doesn't, but, like, on occasion it's like it's done things that would have taken me, you know, a couple hours that it's been great and just, like, solved everything immediately. And if it doesn't, then it's usually easier to verify whether or not the answer is correct than to have written it in the first place. And so you check, you go like, "Well, that's just... You're entirely misguided. Nothing here is right," just like, "I'm not going to do this. I'm gonna go write it myself," or whatever. **Swyx** [16:21] Even for non-tech, I had to fix my irrigation system. I had an old irrigation system, I didn't know how it worked to program it. I took a photo, sent it to Claude, and it's like, "Oh yeah, that's like the RT900. This is exactly..." I was like, "Oh, wow, you know, you know a lot of stuff." **Nicholas Carlini** [16:34] Was it right? It was, it was right? **Swyx** [16:35] Yeah, yeah, it was right. It worked. Um- **Nicholas Carlini** [16:37] Did you compare with OpenAI? **Swyx** [16:38] No. I canceled my- **Nicholas Carlini** [16:39] Did you do Claude first? **Swyx** [16:40] ... OpenAI subscription, so I'm a, I'm a Claude boy. Do you have a way to think about this, like, one-offs software thing? One way I talk to people about it is, like, LLMs are kinda converging to, like, semantic serverless functions, you know? Like, you can say something and, like, it can run the function in a way, and then that's it. It just kinda dies there. Do you have a mental model to just think about how long it should live for and, like, uh, anything like that? **Nicholas Carlini** [17:03] I don't think I have anything interesting to say here, no. **Swyx** [17:05] Yeah. **Nicholas Carlini** [17:05] Uh, I will take whatever tools are available in front of me and try and see if I can use them in meaningful ways, and if the- if they're helpful, then great. And if they're not, then fine. And like, you know, there are lots of people that I'm very excited about seeing all of these people who are trying to make better applications that use these or all these kinds of things, and I think that's amazing. I would like to see more of it, but I, I do not spend my time thinking about how to make this any better. **Swyx** [17:27] What's the most underrated thing in the list? I know there's, like, simplify code, solving boring task, or maybe is there something that you forgot to add that you wanna, uh, throw in there? **Nicholas Carlini** [17:37] I mean, okay, so, so in the list I only put things that people could look at and go, "I understand how this solved my problem." I didn't want to put things where the model was very useful to me but it would not be clear to someone else that it was actually useful. So for example, one of the things that I use it a lot for is debugging errors, but the errors that I have are very much not the errors that anyone else in the world will have. And in order to understand whether or not the solution was right, you just have to trust me on it because, you know, like, I got my machine in a state That like CUDA was not talking to, uh, whatever- **Swyx** [18:16] Right **Nicholas Carlini** [18:16] ... some other thing, the versions were mismatched, something, something, something, and everything was broken. And like, I could figure it out without interacting with the model, and it gave... it, like, told me the steps I needed to take. But at the end of the day, when you look at the conversation, you just have to trust me that it worked. And I didn't want to write things online that were this like, "You have to trust me that what I'm saying." I want everything that I said to, like, have evidence that, like, here's the conversation, you can go and check whether or not this actually solved the task- **Swyx** [18:43] Mm-hmm **Nicholas Carlini** [18:43] ... as I just said that the model does. Because a lot of people, I feel like, say, "I used a model to solve this very complicated task," and what they mean is the model did 10% and I did the other 90%, or something. I, I wanted everything to be verifiable, and so one of the biggest use cases for me, I didn't describe even at all because it's not the kind of thing that other people could have verified by themself. So that maybe is, like, one of the things that I, I wish I maybe had said a little bit more about and just, just stated that the way that this is done because I feel like that, that this didn't come across quite as well. **Swyx** [19:14] Mm-hmm. **Nicholas Carlini** [19:14] But, uh, yeah, of the things that I talked about, the thing that I think is most underrated is the ability of it to solve the uninteresting parts of problems for me right now, where people always say, this is one of the biggest arguments that I don't understand why people say, is the model can only do things that people have done before, therefore the model is not going to be helpful in, in doing new research or, like, discovering new things. And as someone whose day job is to do new things- **Swyx** [19:43] Hmm **Nicholas Carlini** [19:43] ... like, like what is research? Research is doing something literally no one else in the world has ever done before. So, like, this is what I do, like, every single day. 90% of this is not doing something new. Like 90% of this is, like, doing things a million people have done before, and then a little bit of something that was new. There's a reason why we say we stand on the shoulders of giants. It's true. Almost everything that I do is something that's been done many, many times before, and that is the piece that can be automated. Even if the thing that I'm doing as a whole is new, it is almost certainly the case that the small pieces that build up to it are not. And a number of people who use these models, I feel like, expect that they can either solve the entire task or none of the task. But now I find myself very often, even when doing something very new and very hard, having models write the easy parts for me. And the reason I think this is so valuable, everyone who programs understands this, like you're currently trying to solve some problem and then you get distracted. And, you know, what- whatever the case may be, someone comes and talks to you, you have to go look up something online, whatever it is. You lose a lot of time to that, and one of the ways we currently don't think about being distracted is you're solving some hard problem and you realize you need a helper function that does X, where X is like, it's a known algorithm. Any person in the world, you say like, "Give me the algorithm that, you know, have a dense graph or a sparse graph, I need to make it dense. You can do this by, you know, doing some matrix multiplies- **Swyx** [21:08] Yeah, yeah **Nicholas Carlini** [21:08] ... and then you go, "Okay." It's like this is a solved problem. I knew how to do this 15 years ago. But it distracts me from the problem I'm thinking about in my mind. I needed this done. And so instead of using my mental capacity and solving that problem and then coming back to the problem I was originally trying to solve, you could just ask model, "Please solve this problem for me." It gives you the answer. You, you run it, you can check that it works very, very quickly, and now you go back to solving the problem without having lost all the mental state. And I feel like this is one of the things that's been very useful for me. **Swyx** [21:35] And in terms of this concept of, um, expert users versus non-expert users, floors versus ceilings, you had some strong opinion here that like you, you, you basically it, it actually is more beneficial for non-experts. **Nicholas Carlini** [21:46] Yeah, I don't know. I think it could go either way. Let me give you the argument for both of these. **Swyx** [21:51] Yes. **Nicholas Carlini** [21:51] Okay. So I can only speak on the expert user behalf because, you know, I've been doing computers for a long time. And so yeah, the cases where it's useful for me are exactly these cases where I can check the output. I know, and anything the model could do, I could have done. I could have done better. I can check every single thing that the model's doing and make sure it's correct in every way. And so I can only speak and say, say definitely it's been useful for me. But I also see a world in which this could be very useful for the kinds of people who do not have this knowledge, with caveats, because I'm not one of these people, I don't have this direct experience. But one of these big ways that I can see this is for things that you can check fairly easily, someone who could never have asked or have written a program themselves to do a certain task could just ask for the program that does the thing, and, you know, some of the times it won't get it right, but some of the times it will, and they'll be able to have the thing in front of them that they, like, just couldn't have done before. And we see a lot of people trying to do applications for this, like integrating language models into spreadsheets. **Swyx** [22:50] Yeah. **Nicholas Carlini** [22:50] Spreadsheets are, spreadsheets run the, run the world, and there are some people who know how to do all the complicated spreadsheet equations and various things, and other people who don't, who just, like, use the spreadsheet program, but, like, just manually do all of the things one by one by one by one. **Swyx** [23:04] Mm. **Nicholas Carlini** [23:05] And this is a case where you could have a model that could try and give you a solution, and as long as the person is rigorous in testing that the solution does actually the correct thing, and this is the part that I'm worried about most, you know, I think depending on these systems in ways that we shouldn't. Like, this is what my research says. My research says is entirely on this, like you probably shouldn't trust these models- **Swyx** [23:24] Mm-hmm **Nicholas Carlini** [23:24] ... to do the things in every sort of situations. Like I, I understand this very deeply. Um, and so I'm, I think that it's possible for people who don't have this knowledge to make use of these tools in ways, but I'm worried that it might end up in a world where people just blindly trust them, deploy them in situations that aren't, they probably shouldn't, and then someone like me gets to come along and just break everything because everything is terrible. And so, like, I'm, I am very, very worried about that being the case, but I think if done carefully, it is possible that these could be very useful. **Swyx** [23:55] Yeah. There, there is some research out there that shows that when people use LMS to generate code, they, they do generate less secure code. **Nicholas Carlini** [24:02] Yeah. Dan Boneh has a nice paper on this. There are, yeah, there are a bunch of, um- **Swyx** [24:06] Yeah **Nicholas Carlini** [24:06] ... a bunch of papers that, that touch on exactly this. **Swyx** [24:08] My slight issue is, you know, is there an agenda here, like- **Nicholas Carlini** [24:11] I mean, okay, yeah. Um, uh- Dan Boneh at least, uh, the one they have, like I, I fully trust everything that sort of he, yeah, he- **Swyx** [24:16] Sorry, I don't, I don't know who Dan is **Nicholas Carlini** [24:17] ... uh, he, a professor at Stanford. Um, yeah, he, he and some students have some things on this, and, uh, yeah, there's like a number of the... I agree that a lot of the stuff feel like people have an agenda behind it. **Swyx** [24:26] Yeah. **Nicholas Carlini** [24:26] Um, there are some that don't, and, like- ... uh, I sort of, I trust them to, like, have done the right thing. I also think e- even on this, though, we have to be careful because the argument, whenever someone says, "X is true about language models," you should always append the suffix, "For current models," because I'll be the first to admit, I was one of the f- people who was very much on the opinion that these language models are fun toys and are gonna have absolutely no practical utility, and, you know, if you had asked me this, let's say, in 2020, I still would've said the same thing. Like, after I had seen GPT-2, I had written a couple of papers studying GPT-2 very carefully. I still would've told you these things are toys. And when I first read the RLHF paper and the, um, you know, instruction tuning paper, I was like, "Nope, this is like, you know, this thing that these weird AI people are doing. Like, it's like they're trying to make some analogies to people that makes no sense. Like, it's just like I don't even care to read it. I just, like, I saw the, what it was about and just didn't even look at it." I, I was obviously wrong. These things can be useful, and I feel like a lot of people had the same mentality that I did and decided not to change their mind. **Swyx** [25:32] Yeah, yeah, yeah. **Nicholas Carlini** [25:33] And I feel like this is the thing that I want people to be careful about. I want, I want them to at least know what is true about the world so that, that they can then see that maybe they should reconsider some of the opinions that they had from four or five years ago that may just not be true about today's models. **Swyx** [25:47] Mm-hmm. Yeah. Specifically because you brought up spreadsheets, I want to share, like, my personal experience because I think Google's done a really good job that people don't know about, which is if you use Google Sheets, it's, Gemini is integrated in, in, inside of Google Sheets, and it helps you write formulas. **Nicholas Carlini** [26:00] Great. That's news to me. **Swyx** [26:02] Right? Uh, they, they don't, they don't maybe do a good job. Like, if you, unless you watch Google I/O, there was no other opportunity to learn that Gemini is now in your Google Sheets. And so now I just don't write formulas manually anymore. I just prompt Gemini to do it for me, and it does it. **Nicholas Carlini** [26:16] Yeah. One of the problems that these machine learning models have is a discoverability problem. I think this will be- ... this will be figured out. I mean, it's the sa- it's the same problem that you have with, like, any assistant. Like, you're given a blank box, and you're like, "What, what do I do with it?" No, I, I think this is great. These, more of these things, um, it would be good for them to exist. I want them to exist in ways that we can actually make sure that they're done correctly. I don't want to just have them be pushed into more and more things just blindly. I feel like lots of people there are far too many X plus AI, where X is, like, arbitrary thing in the world that has nothing to do with it and could not be benefited at all, and they're just doing it because they want to use the word, and I don't want that to happen. **Swyx** [26:59] You don't want an AI fridge? **Nicholas Carlini** [27:01] No. Um, yes, I do not want my fridge on the internet. I do not want like... Yeah, okay, anyway, let's not go down that rabbit hole. I understand why some of that happens, because people want to sell things or whatever. But I feel like a lot of people see that, and then they write off everything as a result of it, and I just want to say there are allowed to be people who are trying to do things that don't make any sense. Just ignore them. Do the things that make sense. **Swyx** [27:23] Another chunk of use cases was learning, so both explaining code, uh, being a API reference, all of these different things. Any suggestions on, like, how to go at it? I feel like, uh, there, you know, one thing is, like, generate code and then explain to me. One way is, like, just tell me about this technology. Another thing is like, "Hey, I read this online. Kind of help me understand it." Any best practices on getting the most out of it or? **Nicholas Carlini** [27:48] Yeah, I don't know if I have best practices. I have how I use them. **Swyx** [27:51] Yeah. **Nicholas Carlini** [27:51] Uh, yeah, I find it very useful for cases where I understand the underlying ideas, but I have never used them in this way before. I know what I'm looking for, but I just don't know how to get there. And so, yeah, as an API reference is a great example. You know, the, the tool everyone always picks on is, like, FFmpeg. **Swyx** [28:09] Yeah. **Nicholas Carlini** [28:10] No one in the world knows the command line arguments to do what they want. **Swyx** [28:14] Mm. **Nicholas Carlini** [28:14] They, like, make the thing faster. You know, I, I want lower bitrate. **Swyx** [28:18] Yeah, yeah. **Nicholas Carlini** [28:18] Like, dash V, you know, like, but, like, once you tell me what the answer is, like, I can check. **Swyx** [28:22] Mm-hmm. **Nicholas Carlini** [28:22] Like, this, this is one of these things where it's, it's great for these kinds of things. Or, you know, in other, in other cases, things where I don't really care that the answer is 100% correct. So, for example, I do a lot of security work. Most of security work is reading some code you've never seen before and finding out which pieces of the code are actually important- **Swyx** [28:41] Mm-hmm **Nicholas Carlini** [28:41] ... because, you know, most of the program isn't actually do anything to do with security. It has, you know, the display piece or the other piece or whatever, and, like, you just, you would only even ignore all of that. So one very fun use of models is to, like, just have it describe all of the functions and just skim it and be like, "Wait, which ones look like approximately the right things to look at?" Because otherwise, what are you gonna do? You're gonna have to w- read them all manually. And when you're reading them manually, you're gonna skim the function anyway- **Swyx** [29:06] Yeah **Nicholas Carlini** [29:06] ... and not just, and not even figure out what's going on perfectly. Like, you already know that what, when you're gonna read these things, what you're going to try and do is figure out roughly what's going on, then you'll delve into the details. This is a great way of just doing that, but faster because it will abstract most of what is right. It's gonna be wrong some of the time. I don't care. I would've been wrong, too. And as long as you treat it with this way, I think it's great. And so, like, one of the particular use cases I have in the thing is decompiling binaries, where, you know, oftentimes people will release a binary, they won't give you the source code, and you wanna figure out how to, how to attack it. And so one thing you could do is you could try and run some kind of decompiler. It turns out for the thing that I wanted, none existed. And so, like, I spent too many hours doing it by hand before I first thought, you know, like, "Why am I doing this? I should just check if the model can do it for me." **Swyx** [29:52] Mm-hmm. **Nicholas Carlini** [29:52] And it turns out that it can, and it can turn the compiled source code, which is impossible for any human to understand, into the Python code that is entirely reasonable to understand. And, you know, it doesn't run. It has a bunch of problems, but, like, it's so much nicer that it's immediately a win for me. I can just figure out approximately where I should be looking and then spend all of my time doing that sort of by hand. And again, like, you get a, a big win there. ### AGI Speculation **Swyx** [30:12] So, I mean, I fully agree with, uh, you know, all, all those use cases, and, uh, uh, e- especially for you as a security researcher and, and having to dive into multiple things, I imagine that's super helpful. I do think, uh, we want to sort of move to your other blog post, but, you know, I, I wanted to, uh, you, you ended your post with a little bit of a teaser about your next post and your, your speculations. What are you thinking about? **Nicholas Carlini** [30:34] Okay. So I, I want to write something, and I will do that at some point when I, I have time, maybe after I'm done writing my current papers for ICLR or something, where I want to talk about some thoughts I have for where language models are going in the near-term future. The reason why I want to talk about this is because, again, I feel like the discussion tends to be people who are either very much AGI by 2027, or- **Swyx** [30:56] Always five, five, uh, years away. **Nicholas Carlini** [30:58] Yes. Uh, or are going to make statements of the form, you know, "LLMs are the wrong path, and, you know, we should be s- abandoning this, and we should be doing something else instead." And again, I feel like people tend to look at this and see these two polarizing options and go, "Well, those obviously are both very far extremes. Like, how do I actually... L- like, w- what's a more nuanced take here?" **Swyx** [31:18] Mm-hmm. **Nicholas Carlini** [31:18] And so I, I want... I have some opinions about this that I, I want to put down, just saying, you know, I have wide margins of error. I think you should, too. If you would say there's a 0% chance that something, you know, the models will get very, very good in the next five years, you're probably wrong. If you're gonna say there's a 100% chance that in the next five years some- then you're probably wrong. And, like, to be fair, most of the people, if you read behind the headlines, actually say something like this. **Swyx** [31:43] Yeah, yeah, yeah. **Nicholas Carlini** [31:43] But it's very hard to get clicks on the internet- ... of, like, some things may be good in the future. Like, everyone, everyone wants, like, you know, a very, like, nothing is gonna be good. This is entirely wrong. It's gonna be amazing. You know, like, they, they wanna see this. I want things who have, who, people who have negative reactions to these kinds of extreme views to be able to at least say, like, to tell them, "There is something real here. It may not solve all of our problems, but it is probably going to get better. I don't know by how much." And that's, that's basically what I want to say. And then at some point I'll talk about the safety and security things as a result of this, because the way in which security intersects with these things depends a lot in exactly how people use these tools. You know, if it turns out to be the case that these models get to be truly amazing and can solve, you know, tasks completely autonomously, that's a very different security world to be living in than if there's always a human in the loop. And the types of security questions I would want to ask would be very different. And so I think, you know, in s- some very large part, understanding what the future will look like a couple of years ahead of time is helpful for figuring out which problems as a security person I want to solve now. **Swyx** [32:49] You mentioned getting clicks on the internet, but you don't even have, like, an X account or anything. How do you get people to read your stuff? What's the, what's your distribution strategy? Because this post was popping up everywhere. And then people on Twitter were like, "Nicholas Carlini wrote this. Like, what's his handle?" And it's like, he doesn't have it. It's like, how did you find it? What's the story? **Nicholas Carlini** [33:07] Uh, so I have an RSS feed- **Swyx** [33:09] Uh-huh **Nicholas Carlini** [33:09] ... and an email list, and that's it. I don't like most social media things. I feel like, on principle I feel like they have some harms. As a person, I have a problem when people say things that are wrong on the internet. **Swyx** [33:22] Mm-hmm. **Nicholas Carlini** [33:22] And I would get nothing done- ... if I were to have a Tw- I would spend all of my time correcting people and getting into fights. And so I feel like it is just useful for me for this not to be an option. I tend to just post things online. Um, yeah, it's a very good question. I don't know how people find it. I feel like for some things that I write, other people think it resonates with them, and then they put it on Twitter, and- **Swyx** [33:44] Hacker News as well. **Nicholas Carlini** [33:45] Sure, yeah, yeah. I am, because my day job is, like, doing research, I get no value for, like, having this be picked up. Like, there's no, like, whatever. **Swyx** [33:55] Yeah, yeah. **Nicholas Carlini** [33:55] I, I don't need to be someone who, like, has to have this other thing to, like, give talks or, like... And so I feel like I can just say what I want to say, and if people find it useful, then they'll share it widely. You know, this one went pretty wide. I wr- I wrote a thing, whatever, like, sometime late last year about how to recover data off of an Apple Profile drive from 1980. **Swyx** [34:17] Mm-hmm. **Nicholas Carlini** [34:17] Uh, this probably got, I think, like, 1000x less views than this, but I, I don't care. Like, that's not why I'm doing this. **Swyx** [34:23] Yeah, yeah. **Nicholas Carlini** [34:23] Like, like this is the, the benefit of having a thing that I actually care about, which is my research. I would care much more if that didn't get seen. This is, like, a thing that I write because I have some thoughts that I just wanna put down. **Swyx** [34:32] Yeah. I think it's the, the long form thoughtfulness and au- authenticity that is sadly lacking sometimes in, in modern discourse that, uh, makes it attractive, and I, I think now you have a little bit of a brand of, um, you are an independent thinker, writer, person that, you know, people are tuned in to pay attention to what's, whatever's next coming. **Nicholas Carlini** [34:52] Yeah, I mean, this, this kind of worries me a little bit. I don't, like, whenever I have a popular thing that, like, and then I write another thing which is, like, entirely unrelated- **Swyx** [34:59] Oh, yeah **Nicholas Carlini** [34:59] ... like, I, I don't, I don't want people- **Swyx** [35:01] You should actually just throw people off right now. **Nicholas Carlini** [35:02] Exactly. Yeah. I was just like, I'm, I'm trying to figure out, like, I need to put something else online, so, like, the last two or three things I've done in a row have been, like, actually, like- **Swyx** [35:09] Serious things **Nicholas Carlini** [35:10] ... like, things, things that people should care about. **Swyx** [35:10] Yes. **Nicholas Carlini** [35:11] So I, I, I have a couple of things. I'm trying to figure out which one do I put online to just, like, cull the list of people who have subscribed to my email, and so, like, tell them, like, "No, like, what you're here for is not informed, well-thought-through takes. Like, what you're here for is whatever I want to talk about, and if you don't, you don't, you're not up for that, then, like, you know, go away." **Swyx** [35:24] Yeah, yeah. **Nicholas Carlini** [35:24] Like, this, like, this is not what I want out, out of my, my personal website. **Swyx** [35:28] So, like, here's, like, top 10 enemies or something. What's the next project you're gonna work on that is completely unrelated to research, LLMs, or what, what games do you wanna port into the browser next? **Nicholas Carlini** [35:39] Uh, okay, yeah. So maybe, um, okay, yeah, h- here's a fun question. Uh, how much data do you think you can put on a single piece of paper? ### Paper Data **Swyx** [35:47] I mean, if you can think about bits and atoms. **Nicholas Carlini** [35:49] Yeah. Uh, no, like, like normal printer. Like you, like I give you of- office printer. How much data can you put on a piece of paper? **Swyx** [35:54] Can you read a code it? So- **Nicholas Carlini** [35:57] Like a- **Swyx** [35:57] ... if you're like, you know, Base64A or whatever. **Nicholas Carlini** [35:59] Yeah, whatever you want. **Swyx** [35:59] It's, uh- **Nicholas Carlini** [36:00] Like, uh, you, you get normal off-the-shelf printer- **Swyx** [36:01] How many- **Nicholas Carlini** [36:01] ... off-the-shelf scanner. How much data? **Swyx** [36:03] I'll just throw it out there, like, uh, 10 megabytes. **Nicholas Carlini** [36:07] Well, that's enormous. **Swyx** [36:08] I know. **Nicholas Carlini** [36:09] Yeah. That's a lot. **Swyx** [36:10] Like, really small font. **Nicholas Carlini** [36:11] That's my question. **Swyx** [36:12] Yeah. **Nicholas Carlini** [36:13] Uh, so I, I have a thing. It does about a megabyte. **Swyx** [36:15] Yeah, okay. There you go. **Nicholas Carlini** [36:15] So it's- **Swyx** [36:16] That's off by an order of magnitude. **Nicholas Carlini** [36:17] Yeah, okay. So it's about, uh, in particular, it's about 1.44 megabytes. Uh, which is- **Swyx** [36:21] A floppy disk. **Nicholas Carlini** [36:21] Yeah, exactly. This is, this is the post. It'll be the title at some point is, "A floppy disk"- **Swyx** [36:24] "A paper is a floppy disk"? **Nicholas Carlini** [36:25] Yeah. So this is a little hard because, you know, so, uh, you can do the math and you get, uh, eight and a half by 11. You can print a 300 by 300 DPI, and this gives you two megabytes. And so you need to be able... Like, so if you, every single pixel, you need to be able to recover up to, like, 90-plus percent, uh, like, 95%, 90, like 99-point-something percent accuracy in order to be able to, to actually decode this off the paper. This is, uh, the one of the things that I'm considering. Uh, I, I, I need to, like, get a couple more things working for this where, you know, again, I'm, I'm running to some, um, some random problems, but this is probably... This will be one thing that I'm gonna talk about. Uh, there's this contest called the International Obfuscated C Code Contest, which is amazing. People try and write the most obfuscated C code that they can, which is great, and I have a submission for that whenever they, um, they open up the next one for it, and I'll write about that submission. I have a very fun, uh, gate-level emulation of an old CPU that runs, like, fully precisely, and it's, it's a fun kind of thing. **Swyx** [37:20] Yeah, interesting. Your comment about the, the piece of paper reminds me of when I was in college, and you would have, like, one cheat sheet that you could write. **Nicholas Carlini** [37:27] Right. **Swyx** [37:27] Right? So you have a formula, a theoretical limit for bits per in, in, uh, inch, and, uh, you know, is it... That's how much I would squeeze in- **Nicholas Carlini** [37:34] Yeah, yeah, yeah **Swyx** [37:34] ... really, really small. **Nicholas Carlini** [37:35] No, that, right. **Swyx** [37:36] So on those sheets. **Nicholas Carlini** [37:36] This is definitely, yeah. **Swyx** [37:37] Okay. We are also gonna talk about your benchmarking because you released your own benchmark- ### Personal Benchmarks **Nicholas Carlini** [37:42] Sure **Swyx** [37:42] ... that got some attention, uh, thanks to some friends on the internet. What's the story behind your own benchmark? Do you not trust the open source benchmarks? What's going on there? **Nicholas Carlini** [37:52] Okay. Benchmarks tell you how well the model solves the task the benchmark was designed to solve. For a long time, models were not useful, and so the benchmark that you tracked was just something someone came up with because you need to track something. All of deep learning exists because people tried to make models classify digits and classify images into 1,000 classes. There is no one in the world who cares specifically about the problem of distinguishing between 300 breeds of dog for an image that's 224x224 pixels, and yet, like, this is what drove a lot of progress. And people me- did this not because they cared about this problem, because they, they wanted to just measure progress in some way, and a lot of benchmarks are o-are of this flavor. You want to construct a task that is hard, and we will measure progress on this benchmark, not because we care about the problem per se, but because we know that progress on this is in some way correlated with making better models. And this is fine when you don't want to actually use the models that you have. But when you want to actually make use of them, it's important to find benchmarks that track with whether or not they're useful to you. And the thing that I was finding is that there would be model after model after model that was being released that would find some benchmark that they could claim state-of-the-art on and then say, "Therefore, ours is the best," and that wouldn't be helpful to me to know whether or not I should then switch to it. So the argument that I tried to lay out in this post is that more people should make benchmarks that are tailored to them. And so what I did is I wrote a domain-specific language that you p- anyone can write for and say they can... You can take ch- tasks that you have wanted models to solve for you, and you can put them into your benchmark that's the thing that you care about, and then when a new model comes out, you benchmark the model on the things that you care about, and you know that you care about them because you've actually asked for those answers before. And if the model scores well, then you know that for the kinds of things that you have asked models for in the past, it can solve these things well for you. This has been useful for me because when another model comes out, I just, I can run it. I can see, is this, does this solve the kinds of things that I care about? And sometimes the answer is yes, and sometimes the answer is no. And then I can decide whether or not I want to use that model or not. I don't want to say that existing benchmarks are not useful. They're very good at measuring the thing that they're designed to measure. But in many cases, what that's designed to measure is not actually the thing that I want to use it for, and I, I would expect that the way that I want to use it is different than the way that you want to use it, and I would just like more people to have these things out there in the world. And the final reason for this is it is very easy, if you want to make a model good at some benchmark, to make it good at that benchmark. You can sort of like find the distribution of data that you need and train the model to be good on the distribution of data, and then you have your model that can solve this benchmark well. And by having a benchmark that is not very popular, you can be relatively certain that no one has tried to optimize- **Swyx** [40:39] Mm-hmm **Nicholas Carlini** [40:39] ... their model for your benchmark. And I would like there to be- **Swyx** [40:41] So publishing your benchmark is a little bit- **Nicholas Carlini** [40:43] Okay, sure. **Swyx** [40:44] ... counterintuitive. **Nicholas Carlini** [40:45] Yes, okay. So, so, um, so my, my hope in doing this was not that people would use mine as theirs. **Swyx** [40:51] Yes. **Nicholas Carlini** [40:51] My hope in doing this was that people would say- **Swyx** [40:53] You should make yours. **Nicholas Carlini** [40:53] Yes. You should make your benchmark and, you know, if for example, you know, there were even a s- very small fraction of people, 0.1% of people who made a benchmark that was useful for them, this would still be hundreds of new benchmarks that not want to make one myself, but I might want to... I might know the kinds of work that I do is a little bit like this person, a little bit like that person. I'll go check what the, how it is on their benchmarks, and I'll see, like, it's sort of roughly I'll, I'll get a good sense of what's going on. Because the alternative is people just do this, like, vibes-based evaluation thing where it's like the, you interact with the model five times, and you see if it worked on the kinds of things that you just, like, your toy questions. But, like, five questions is a very low bit output from, like, whether or not it works for this thing. And if you could just automate, you know, running it 100 questions for you, like, this, it's a, it's a much better evaluation. So that's, that's why I did this. **Swyx** [41:37] Yeah. I like the idea of going through your chat history and actually pulling out real life examples. I regret to say that I don't think my chat history is used as much these days because I'm using Cursor, like, the, you know, the sort of I- native AI IDE. So your examples are all coding related and, you know, the, the immediate question is, like, now that you've written the how I use AI post, which is a little bit broader, is, are you able to translate all these things to evals? Are some things unevaluatable? **Nicholas Carlini** [42:04] Right. A number of things that I do are harder to evaluate. So this is the, the problem with a, with a benchmark is you need some way to check whether or not the output was correct. And so all of the kinds of things that I can put into the benchmark are the kinds of things that you can check. You can check more things than you might have thought would be possible if you do a little bit of work on the back end. So for example, all of the code that I have the model write, it runs the code and sees whether the answer is, is the correct answer. Or in some cases, it runs the code, feeds the output to another language model, and the language model judges was the output correct. And again, is using a language model to judge here perfect? No. But, like, what's the alternative? The alternative is to not do it. And what I care about is just is this thing broadly useful for the kinds of questions that I have? And so as long as the accuracy is better than roughly random- Like, I'm, I'm okay with this. I've inspected the outputs of these, and, like, they're almost always correct. So if you sort of... If you ask the model to judge these things in the right way, they're, they're very good at being able to tell this. And so, yeah, I, I probably think this is a useful thing for people to do. **Alessio** [43:04] You complain about prompting and being lazy and how you do not wanna tip your model, and you do not wanna murder a kitten just to get the right answer. How do you see the evolution of, like, prompt engineering? Even, like, 18 months ago, maybe, you know, was kinda, like, really hot, and people wanted to, like, build companies around it. Today, it's like the models are getting good. Do you think it's gonna be less and less relevant going forward, or what's the minimum viable prompt? **Nicholas Carlini** [43:29] Yeah, I don't know. I feel like a big part of making an agent is just, like, a fancy prompt that, like, you know, calls back to the model again. I have no opinion. It seems like maybe it turns out that this is really important. Maybe it turns out that this isn't. I guess the only comment I was making here is just to say oftentimes when I use a model and I find it's not useful and I talk to people who, who help make it, the answer they usually give me is, like, "You're using it wrong." Which, like, reminds me very much of, like, the you're holding it wrong from, like, the iPhone kind of thing, right? Like, you know, like, I don't care that I'm holding it wrong. I'm holding it that way. If the thing, if the thing is not working with me, then, like, it's not useful for me. Like, it may be the case that there exists a way to ask the model such that it gives me the answer that's correct, but that's not the way I'm doing it. **Alessio** [44:10] Yeah. **Nicholas Carlini** [44:11] If I have to spend so much time thinking about how I want to frame the question, that it would've been faster for me just to get the answer, it didn't save me any time. And so oftentimes, you know, what I do is, like, I just dump in whatever current thought that I have in whatever ill-formed way it is, and I expect the answer to be correct. And if the answer is not correct, like, in some sense, maybe the model was right to give me the wrong answer. Like, I, I may have asked the wrong question. Well, but I want the right answer still. And so, like, I just wanna sort of get this as a thing, and maybe the way to fix this is you have some default prompt that always goes into all the models or something, or you, you do something, like, clever like this. And it would be great if someone had a way to package this up and make a thing. I think that's entirely reasonable. Maybe it turns out that as models get better, you don't need to prompt them as much in this way. I, I don't know. I just wanna use the things that are in front of me. **Alessio** [44:56] Do you think that's, like, a limitation of just how models work? Like, you know, at the end of the day, you're using the prompt to kinda, like, steer it in the latent space. Like, do you think there's a way to actually not make the prompt really relevant and have the model figure out? Or, like, what's the- **Nicholas Carlini** [45:09] I mean, you could fine-tune it into the model, for example, that, like, it's supposed to... I mean, it seems like some models have done this, for example. Like, some recent model, many recent models, if you ask them a question, computing an integral of this thing, they'll say, "Let's think through this step by step." And then they'll go through a step-by-step answer. I didn't tell it. Two years ago- **Alessio** [45:24] Mm-hmm **Nicholas Carlini** [45:24] ... I would have had to have prompted it. **Alessio** [45:26] Yeah. **Nicholas Carlini** [45:26] Think step by step when solving the following thing. Now you ask them the question, and the model says, "Here's how I'm gonna do it. I'm gonna take the following approach," and it, like, sort of self-prompts itself. Is this the right way? Seems reasonable. Maybe you don't have to do it. I don't know. Uh, this is for the people whose job is to make these things better, and yeah, I, I just wanna, uh, to use these things. **Alessio** [45:43] Yeah. **Swyx** [45:43] Yeah. For listeners, that would be Orca and AgentInstruct w- uh, is the SOTA on this stuff. **Nicholas Carlini** [45:48] Great. **Swyx** [45:49] Yeah. **Alessio** [45:49] Does a few-shot is included in the lazy prompting? Like, do you do few-shot prompting? Like, do you collect some examples when you wanna put them in, or? **Nicholas Carlini** [45:57] I don't because, uh, usually when I want the answer, I just- **Alessio** [46:01] Yeah, yeah **Nicholas Carlini** [46:01] ... I want to get the answer and then go like, "Yeah, like"- **Swyx** [46:03] Brutal. This is hard mode. **Nicholas Carlini** [46:04] Yeah, exactly. But, like, this is, this is fine. Like, you know, I want to be clear, like, this, there's a difference between testing the ultimate capability level of the model- **Swyx** [46:10] Yeah **Nicholas Carlini** [46:11] ... and testing, you know, the thing that I'm doing with it. What I'm doing is I'm not exercising its full capability level because, like, there are almost certainly better ways to ask the questions and sort of really see how good the model is. And if you're evaluating a model for, like, being state-of-the-art, like, this is ultimately what we care about. And so I'm entirely fine with people doing fancy prompting to show me what the true capability level could be because it's really useful to know what the ultimate level of the model could be. But I think it's also important just to have available to you how good the model is if you don't do fancy things. **Swyx** [46:39] Yeah. I, I would say here's the divergence between how models are marketed these days versus how people use it, which is when they test MMLU, they'll do, like, five shots, 25 shots, 50 shots, and no one's providing 50 examples. **Nicholas Carlini** [46:53] I, I completely agree. Um, you know, for these numbers, the problem is everyone wants to get state-of-the-art on the benchmark, and so you find the way that you can ask the model the questions so that you get state-of-the-art on the benchmark. And it's good... Like, it's legitimately good to know. Like, it's good to know the model can do this thing if only you try hard enough because it means that if I'm, if I have some task that I want to be solved and, like, know what the capability level is, and I could get there if I was willing to work hard enough. And the question then is, should I work harder in figuring out how to ask the model the question, or do I just do the thing myself? And for me, like, you know, I have programmed for many, many, many years. It's often just faster for me just to do the thing than to, like, figure out the incantation to ask the model. But I can imagine someone who has never programmed before might be fine writing five paragraphs in English describing exactly the thing that they want and have the model build it for them if the alternative is not. But again, uh, this goes to all these questions of how are they going to validate, should they be trusting the output, these kinds of things. But yeah. **Swyx** [47:50] One problem with your eval paradigm, and most eval paradigms, it's not, I'm not picking on you, is that we're actually training these things for chat, for interactive back and forth, and you actually obviously reveal much more information in a, in the same way that asking 20 questions, uh, reveals more information in sort of like a tree search branching sort of way. Then this is also, by the way, the problem with LMSYS arena, right? Where, where the vast majority of prompts are single question, single answer eval, done. But actually, the way that we use chat things, in the way, even in the, the stuff that you posted in your how I use AI stuff, you have maybe 20 turns of back and forth. How do you eval that? **Nicholas Carlini** [48:26] Yeah. Um, okay, very good question. This is a thing that I think many people should be doing more of. I would like more multi-turn evals. I might be writing a paper on this at some point if I get around to it. A couple of the evals in the benchmark thing I have are already multi-turn. You mentioned 20 questions. I have a 20 question eval there just for fun. Um, but I have a couple others that are like, I just tell the model, um, "Here's my Git thing. Figure out how to cherry-pick off this other branch and move it over there." And so what I do is I just, I basically build a tiny little agenty thing. I just ask the model how I do it. I run the thing on Linux. I spin up a dock- this is what I want a Docker for. I spin up a Docker container. I run the con- the, whatever the model told me the output is to, to do is. I feed the output back to the model. I repeat this many rounds, and I, then I check at the very end, you know, does the g- git commit history show that it is correctly cherry-picked in this way? And, you know, so I, I have a couple of these. I agree that I have many fewer than what I actually use them for, and I think the reason why is just that it's hard to evaluate this. Like, it's, it's more challenging to do this kind of evaluation. Yeah, I would like to see a lot more of these kinds of things to exist so that people could come up with these evals that more closely measure what they're actually doing. **Alessio** [49:35] Just before we wrap on this, there was one example about a UU encode, and, uh, you mentioned how, like, nobody uses this thing anymore. When you run into something like this, and you know that no more data is gonna get produced on this thing, do you figure out how to, like, fine-tune the model? Like, do you, if it really mattered to you, put together some examples, or would you just say, "Hey, the model just doesn't do it, whatever. Move on"? **Nicholas Carlini** [49:58] Yeah. Um, yeah, okay, this was, um, an example of a thing where I was looking at some data f- that was ... It was a file that was produced in, like, the mid-'90s, early '90s or something when UU encoding was actually a thing that people would do. And I wanted the model to be able to automatically determine the type of file to decompress in something, and, like, it was, it was doing it correctly for, like, 99% of cases, and, like, it found a few UU-encoded things where, like, it couldn't figure out this was UU encoding, not Base64. Okay. This is not important. I just was curious if it could do it, and so I put this as a thing. Um, I think probably this is a thing that if you really cared about this task being solved well, you would train a model for, but again, this is one of these kinds of tasks that this was so for some du- dumb project that, like, no one's gonna care about. I just wanted to see if I could do it. If the model was good enough that it gets me 90% of the way there, good. Like, done. Like, I, I figured it out. Like, I can, I can sort of have fun for a couple hours and then move on, and that's all I want. I was not... Like, if, if, if I ever had to train a thing for this, I just was not gonna do it. And so it did well enough for me that I, I could move on. **Swyx** [50:58] It does give me an idea for, um, adversarial examples inside of a benchmark that are basically canaries for over-training on the benchmark. Typically, right now, benchmarks have canary strings, where if you ask it to repeat back the string, and it does, then it's trained on it. But, you know, it's easy to filter out those things. But the benchmarks, y- you put in some things, some questions that are intentionally wrong, and if it gives you the intentionally wrong answer, then you know it's **Nicholas Carlini** [51:20] Yeah. There are actually a couple of, um- **Swyx** [51:22] Oh **Nicholas Carlini** [51:22] ... papers that don't, don't do exactly this but that are doing dataset inference. So the field of, of work called membership inference, this is one of the things I do research on, uh, that tries to figure out did you train on this example or not? **Swyx** [51:34] Yeah. **Nicholas Carlini** [51:34] There's a field called, like, dataset inference. Did you train on this dataset or not? And there's, like, a specific subfield of this that looks specifically at, like, did you train on your test set, or you train on your training- **Swyx** [51:45] Yeah **Nicholas Carlini** [51:45] ... training set? **Swyx** [51:45] Very important. **Nicholas Carlini** [51:46] And they basically look at exactly this. Like, for example, one, there's this paper by Tatsu out of Stanford where they check if the order that the specific questions- **Swyx** [51:56] Yes **Nicholas Carlini** [51:56] ... happen to be in- **Swyx** [51:56] I read that one, yeah **Nicholas Carlini** [51:57] ... matters. **Swyx** [51:57] Yep. **Nicholas Carlini** [51:57] And if the answer is yes, then you probably trained on it because the order of the questions is arbitrary and shouldn't matter. There are a number of papers that follow up on this and do some similar things. I think this is a great, uh, you know, way of doing this now. It might be even better if some people included some canary questions in their benchmarks, but even if they don't, you can already sort of get, start, start getting at this now. **Swyx** [52:13] Yeah. **Alessio** [52:14] Yeah. Let's go into some of your research. I always loved, uh- ### Security Attacks **Nicholas Carlini** [52:17] Sure **Alessio** [52:17] ... security work. I was at Black Hat last week. I had to miss DEF CON. Let's start from the LAION-400M, kinda like data, possible- **Nicholas Carlini** [52:27] Sure **Alessio** [52:27] ... data poisoning. **Nicholas Carlini** [52:27] Okay. **Alessio** [52:27] So basically, the idea is, you know, LAION-400M is one of the biggest image datasets for image models, and a lot of the image gets pulled from live domains. **Nicholas Carlini** [52:36] All of them. **Alessio** [52:36] So it's not all... Yeah. The, the- **Nicholas Carlini** [52:38] E- e- every image gets pulled from a live domain, yes. **Alessio** [52:40] So it's not all stored, and a bunch of the domains expired. So then you went on, and you bought the domains, and you could have put literally anything- ... on it. **Nicholas Carlini** [52:47] Yes. **Alessio** [52:47] And you could have, you could have poisoned every single model that was training- **Nicholas Carlini** [52:50] Yeah **Alessio** [52:50] ... on the dataset. **Nicholas Carlini** [52:51] Yep. It was a lot of fun. **Alessio** [52:53] Maybe just talk about some of the things that people don't think about when it comes to, like, the datasets. We talked before about low background tokens, so before maybe 2020, you can imagine most things you get from the internet a human wrote- **Nicholas Carlini** [53:05] Origin **Alessio** [53:05] ... or like, you know, uh, after 2021, you can imagine most things written are, like, somewhat AI-generated. Any other fun stories or, like, maybe give more of the LAION background. How did you figure out... Di- do you just, like, check all the domains in it and see what expire? Why do they not do it to prevent this? **Nicholas Carlini** [53:21] Yeah. Um, okay. So, so, so why did this paper happen? The adversarial machine learning literature for a very long time was focused on what could I do in the worst case? Because no one was using these tools, and so no one's using them, like, it doesn't make sense to really ask, like, "How do I attack this actual system?" **Alessio** [53:39] Mm-hmm. **Nicholas Carlini** [53:39] And so people would write papers, I mean, me included, I have lots of these, that, like, assume an adversary could do the following, and then list 10 unrealistic things. Then very bad harm could happen. And in some sense, like, you have to do this. If you have no real system in front of you, like, what are you gonna do as a security researcher? One thing you can do is just nothing. You could just wait. But, like, this is a bad option because eventually someone's gonna use these things, and you would rather have a head start. So how do you get a head start? You make a guess. You say, "Maybe future systems will do X." And then y- then you write a paper that sort of looks at this. And then maybe it turns out that some of these are directionally correct, some are not, and so, okay. So, so this has happened for quite some long time. And then machine learning started to work. And the thing that bothered me is it seems like the adversarial machine learning community didn't then try and adapt and try and actually start studying real problems. So we very deliberately started looking, like, what are the problems that actually arise in real systems as they exist now? Like, what is the kind of paper that I could imagine writing that would be at Black Hat? That, like, a real security person would want to, to see, not because here's a fun thing that you can make this machine learning model do, but because legitimately the easiest way to make the bad thing happen is to go after the machine learning model. So the way we decided to do this is, like, sort of a very, like, every time you see some new thing, you say, "Well, here are the bad things that could happen. You know, I could try and do an evasion attack at test time. I could try and do a poisoning attack that made the model train on bad data. I could try and steal the model. I could try and steal the data." You know, the, the list of, like, 10 bad things you could try and make happen, and every time you see some new thing, you ask, "Okay, here's my list of 10 problems Which of them are most important and relevant to this? And you just do this for every single one on the list, and, you know, most of the time the answer is nothing, and you just-- then you get nothing out of it. But, like, on occasion, you sort of figure out, okay, here's this new dataset. It is being distributed in such a way that anyone in the world can buy domains that let you then inject arbitrary images into the dataset. There's the attack. And, like, you know, this is, I think, the way that we, we came to doing this from this motivation of let's try and look at some, some real security stuff. **Swyx** [55:33] I think when people think of AI security, they either think of jailbreaks, you know, which is kinda, like, very limited, or they kinda go the broader, "Oh, is AI gonna kill us all?" I think you've done a lot of awesome papers on, like, the in-between. So one thing is the jailbreak. Like, you've also had a paper on stealing part of a production LLM. You extracted, like, the, uh, Babbage and Ada, like, uh, dimension layers from, like, the OpenAI API. So there's even things that, like, as a user, you're worried about the jailbreaks, but, like, as a model provider, you're actually worried about the- **Nicholas Carlini** [56:05] Yeah, exactly. **Swyx** [56:05] Yeah. **Nicholas Carlini** [56:05] And this paper is, again, with the exact same motivation. Uh, so as some history, there's this field of research called model stealing. Uh, what it's interested in is, um, you have your model that you have trained. It was very expensive. I want to query your model and steal a copy of the model so that I have, I have your model without paying for the training costs. Um, and we have some very nice work that shows that this is possible. Like, I can steal your exact model. As long as your model has, let's say, a couple thousand neurons evaluated in float sixty-four with ReLU-only activation, fully connected networks, I see the full lo- logic outputs, and I can feed in arbitrary floating point sixty-four numbers and inputs. Each of these assumptions I just said is false in practice. **Swyx** [56:42] Mm-hmm. **Nicholas Carlini** [56:42] Like, none of these things are things you can really do. I think it's fun research. I mean, there's a reason, like, w- the paper's at Crypto. Like, the reason it's at Crypto and not, like, at, like, an actual s-security conference because, like, it's a very theoretical kind of thing, and I think it's, like, an important direction for people to think about because maybe you can extend these to make it be possible. But I also think it's worth thinking about the problem from the other direction. Like, let's look at what the real models we have in front of us are, and let's see how we can make those models be vulnerable to stealing attacks. And then we can push from the other direction of, like, let's take the most practical attacks and make them more powerful, and that's again, like, what we're trying to do here. We sort of looked at what, what APIs do actually people expose in the biggest models? How can we use some of that to do as much stealing as we possibly can? Yeah, and f- and for this, um, we ran the attack that le-let us, yeah, stole several of OpenAI's models, um, with their permission. You know, it's sort of it's a fun email to send, you know. "Hello, Mr. Lawyer." Like, um, I saw him at Google. Um, you know, I first have to email the- **Swyx** [57:34] Yeah. **Nicholas Carlini** [57:34] "Hello, Hello, Google Lawyer." **Swyx** [57:35] Oh, God. **Nicholas Carlini** [57:36] "I would like to steal OpenAI's models." And they say, like, you know, "Under no circumstances." And you say, "Okay, but what if they agree to it?" And you're like, "If they agree to it, fine." And you sort of, then you say, "I, I know some people there. I email them like, 'Can I steal your model?'" And they're like, "As long as you delete it afterwards, okay." And I'm like, "Can, can you get your general counsel to put that in writing?" And they're like, "Sure." So, like, sort of we, we, we had all of the lawyers talk to each other. Everyone agreed that, like, you know, it's important to do this. Like, you know, you don't want to actually, uh- **Swyx** [58:02] Yeah **Nicholas Carlini** [58:02] ... you know, um, uh, sort of cause harm when doing security work. And so we got all, all of the, like, the agreements out of the way, and then we went and ran the attack. And yeah, and it worked great, and then we can, we can write the paper. Before we put the paper online, um, we notified everyone who was vulnerable to this, this attack. Some Google models were vulnerable. Some OpenAI mo-models were vulnerable. There were one or two other people who were vulnerable that we didn't name in the paper. We notified them all, gave them ninety days to fix it, which is, like, a standard disclosure period in security. That was all patched. Um, you know, OpenAI got rid of some APIs, and then we put the paper online. **Swyx** [58:33] The fix was just don't show logits. **Nicholas Carlini** [58:35] Yeah. So the fix in particular was don't show log probs when you supply a logic bias, and what you don't show is the logic bias plus the log prob, which is, like- **Swyx** [58:45] Mm-hmm **Nicholas Carlini** [58:45] ... a very narrow thing. They sort of did the narrow thing to prevent this. S-some people were unhappy, but, like, this is, you know, this is the nature of making you, you can have a more useful system or a more secure system in many ways. I really like this example because for a very long time, nothing about GPT-4 would be at all different if the field of, like, the entire field of adversarial machine learning disappeared. Like, everything to do with adversarial examples, like, all of l-l-like, for the most part, like, GPT-4, like, would exist ex-identically. This is not true in other fields in, you know, in system security. Like, the way we design our processors today is fundamentally different because of the security attacks that we've had in the past. You know, the way we design databases, the way we design the internet is fundamentally different because of the way, the, the attacks that we have. And what that means is it means that the attacks that we had were so compelling to the non-security people that they were willing to change and make their systems less useful in order to make the security better. In adversarial machine learning, we didn't have this. We didn't have attacks that were useful enough that you could show it to someone who actually designed a real system, and they'd be willing to say, "I am going to make my system less useful because the attack that you've presented to me is so compelling that I will break the functionality of my system." And this is one of the first cases I think that we, we were able to show this, is someone-- we had an attack that someone said, "I agree with this attack. It's sufficiently bad that I will break utility in order to prevent this attack. And I would like to see more of these kinds of attacks, not because I, I want things to be worse, but because I want to be sure that we have exhausted the space of possible attacks so that it's not gonna be the case that someone else comes up with a very bad thing that, like, they're n-not gonna disclose, sit on for, you know, a couple months, and then go and, and bang on everything and see what they can hit." And this is the hope of, of doing this research direction. **Swyx** [1:00:20] I wanna spell it out for people who are maybe not so specialized in this. Your attack could potentially steal the entire projection matrix? **Nicholas Carlini** [1:00:27] Yeah. W-so, so a model has many layers. We pick one of the layers, and we show how to steal that layer. **Swyx** [1:00:32] Yeah. And then just scaling it up, you can steal the others. **Nicholas Carlini** [1:00:36] For this attack, I do not know. **Swyx** [1:00:37] Yeah. Okay. **Nicholas Carlini** [1:00:38] So this, this is the, this is the important detail. We only steal one in the attack that as we present it, we only know how to steal one layer. For the other research we have done in the past, we have shown how after stealing one layer, you can then extend it to the second layer, and then the second to the third, and the third to the fourth, and you can do this, like, arbitrarily deep. And, you know, we, we have done this in the past, but that made ridiculous assumptions. And what we're trying to do now is similar kind of thing, but let's make f-less ridiculous assumptions. **Swyx** [1:01:03] Yeah. It's kind of like in security how you have, like, privilege escalation. Once you, once you're in the system, you can- **Nicholas Carlini** [1:01:08] Yeah. Yeah, that, that's the hope. And so, like, the, the reason why we want to write these kinds of papers is to say, "Let's always know what the best attack is. Let's have the best attack be public so that people can at least prevent what is, what the best is that, that is known right now." And if someone else were to discover a stronger variant, I would hope that they would take a similar approach, let everyone know how to patch it, patch the thing, release it to everyone, and go from there. **Swyx** [1:01:29] We do also serve people building on top of models, and one thing that I think people are interested in is prompt injections, prompt security, that kind of stuff. I feel like in, in the relevant version of yours, your thing is can I steal the RAG corpus that it might be proprietary to a company? Uh, I don't know if you've heard. **Nicholas Carlini** [1:01:46] No, this is, um, this is a very good question. Um, yeah, there's a f- so, like, there's two kinds of stealing. There's model stealing and there's data stealing. Data stealing is exactly, uh, this kind of question, and I think this is a very good question. In many ways, the answer is, is yes. Even without RAG, you can often steal data that a model was trained on. So we've done some work where we have trained a model, or we, we have shown that for production models... Okay, in this case, um, in the, the most extreme variant, we showed a way to recover training data from GPT 3.5- **Swyx** [1:02:15] Yeah **Nicholas Carlini** [1:02:15] ... Turbo. Yeah, one of my coauthors, Milad, uh, was working on some other random experiments, and he figured out that if you prompt ChatGPT to repeat a word forever, then it will repeat the word many, many, many times in a row and then, like, explode and, like, just start doing random stuff. And when it was doing random stuff, maybe a small percent of the time, maybe 2% of the time, it would just repeat training data back to you, which- **Swyx** [1:02:38] Mm **Nicholas Carlini** [1:02:38] ... you know, is very confusing, but, like, you know, this is a, a thing that, like, happened and was, like, an exciting kind of thing and, like, we've, we've seen this in the past, yeah. **Swyx** [1:02:45] Do we know that is, like, is it exactly the training data- **Nicholas Carlini** [1:02:48] Yes **Swyx** [1:02:48] ... or is it something that looks like the- **Nicholas Carlini** [1:02:49] Identical to the training data- **Swyx** [1:02:51] Because it cannot memorize, it doesn't have the weights to memorize all the training data. **Nicholas Carlini** [1:02:55] No, it can't memorize all the training data, no, definitely. But, um, it can memorize some of it. How am I so certain? We found text that was on the internet, 10 terabytes of data, and what I can say is that the output of the model was a verbatim, at least 50-word-in-a-row match to some other document that appeared on the internet previously. So, all right, there's two possible explanations for this. One is the model happened to come up with the same 50-word-in-a-row sequence as was existed on the internet previously. In principle, this is possible, or it memorized it. And for some of them we have, like, you know, like, several hundred words in a row where, like, you're like, the probability is, like, astronomically low. ### Motivation **Swyx** [1:03:30] So you also have a blog post about why I attack. Last week we did a Man Versus Machine event at Black Hat with, uh, our friend, HD Moore. Um, it was basically like a AI CTF. And then Vijaya, who is the CISO of DeepMind, he also came to the awards ceremony, and I was talking to him. I told him we're gonna interview you, and he was like, "You should ask Carlini why he does not want to build defenses." And so he told me to ask you that. **Nicholas Carlini** [1:03:54] Sure. **Swyx** [1:03:54] So I'll just open the floor to you now to answer. **Nicholas Carlini** [1:03:56] You asked his boss for a question. **Swyx** [1:04:01] Yeah. **Nicholas Carlini** [1:04:01] Yeah, okay, no, so, um, okay, this is a good question. There are a couple of reasons. The most basic level, I attack things because I think it's fun. I feel like people should do things that they find are, are interesting in the world. I also think that it's important to attack things because you don't know what's secure unless you know what the best attacks are, and so it's worth having what the best attacks are in order to be able to discover what is secure. People then say, "Both of these things are true, and yet you should still build defenses." You know, I, I have gotten this a lot, uh, through my, my career. And it is possible that I would be able to construct defenses. You know, on rare occasions I have helped write papers that have defenses. I just don't find it very fun. I have a hard time motivating myself to work on it. And I think this is very important because let's suppose that you w- you decide, "Okay, I am going to be a person who is going to try and do maximal good in the world." Presumably, there are jobs you could take that would, like, save more lives than what you're doing right now. But if you would wake up every day hating your life, it is very unlikely you would do an actually good job. You know, like, I could sort of switch now to be a, a doctor or, you know, to do elderly care or something like this. But someone who actually went into it for the right motivations is going to do so much better than if I just decided, like, "I am going to be a robot. I'm going to ignore what I actually enjoy, and I'm going to do the things that are someone ha- else has described objectively as, like, better for the world." I don't actually think that you would do that good because you're not gonna wake up every morning being like, "I'm excited to solve this problem." You'll do your job from 9:00 to 5:00, and then you'll go home and work on what you actually find fun. And a big part of doing high-quality work is actually being willing to think about these kinds of problems all the time. **Swyx** [1:05:43] Mm-hmm. **Nicholas Carlini** [1:05:43] And whenever, like, a, a new thing comes up, like, you, you want to do the thing. You want to, like, be like, "I have to go to sleep now even though I want to be working on this problem." Like, you will, you will do better work in the grand scheme of things if you sort of look at the product of, you know, how valuable a thing is multiplied by how much you're gonna actually be able to do for it. And there are some, lots of things that will be very high impact that, like, you are just not the right person to solve, and I feel like that's the case for me for defenses, is I, I really just don't care. **Swyx** [1:06:09] Right. **Nicholas Carlini** [1:06:09] Like, I just, like, it's, it's not interesting to me. I, I don't know why. I, I've tried. In order to graduate, my thesis had to have a piece of it which was a defense. And so, so it's there. But, like, that last, you know, little while I was just, I was not having a good time. **Swyx** [1:06:22] Yeah. **Nicholas Carlini** [1:06:22] Like, I, I... it's there. It, like, didn't become a paper. It's, like, a chapter in my thesis, and so I have my PhD, but, like, it's not, like, a thing that, like, actually motivated me- **Swyx** [1:06:30] Yeah **Nicholas Carlini** [1:06:30] ... to, like, be excited by the thing. And so I think maybe some people can get motivated and, and on, and work on things that, like, are really important, and then they should do that. But I feel like if there are things in the world that in principle you could do more good but, like, you're just not the right person for them, you will likely end up doing less good because you will not actually be able to do as much as you really could have if you had tried to do better. **Swyx** [1:06:56] Awesome. Anything else we missed? Any underrated work that you really want people to check out? When... Anything? **Nicholas Carlini** [1:07:03] I mean, no. I mean, like, the, yeah, I, I, I tend to do a fairly broad set of things, so anything you've missed, almost certainly yes. Anything that's particularly important that you have missed, probably not. I feel like, you know, just it's, um, I think people should work on more fun things. **Swyx** [1:07:15] Thank you so much for coming on. **Nicholas Carlini** [1:07:16] Yeah. Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.