# Breaking AI to Fix It: Ian Webster's Journey from Discord's Clyde to Promptfoo's $18M Series A Latent Space · 2025-10-24 Ian Webster, founder of Promptfoo (recently $18M Series A from Thrive Capital and NEA), explains how his journey from building Discord's AI chatbot Clyde to creating the leading AI security testing platform revealed that while evals are table stakes, the real value lies in pre-deployment red teaming that finds application-specific risks. Webster argues that traditional guardrails are insufficient—'you can't fix stupid'—and that Promptfoo uses AI-versus-AI techniques to generate tailored attacks for each app's business context. Over 10% of Fortune 500 companies now use Promptfoo, and he discusses the evolving challenges of MCP security (mostly 'glorified API wrappers') and the need for IDE plugins and code-aware discovery to keep up with increasingly complex agents running 30-minute tasks. The episode also covers why security sales in AI is uniquely hard, how open source builds trust in a space full of 'snake oil,' and why the future of AI security must shift from runtime protection to earlier testing in CI/CD. ## Questions this episode answers ### How does Promptfoo generate tailored attacks for AI applications? Ian Webster demonstrates an AI-vs-AI approach: you provide your application endpoint and business context—its purpose, tools, user roles—and Promptfoo's plug-ins produce attack objectives specific to that context. An attacker LLM then communicates with your application, while a judge evaluates the responses, iterating hundreds of times per objective to find vulnerabilities beyond static prompts, such as unauthorized data access or tool misuse. [12:21](https://addtry.com/9ca4c648-b223-4d97-a551-d60a1e0ec614?t=741000) ### Why did Promptfoo shift from an evals tool to an AI security red teaming platform? Ian explains he saw evals as a commodity, with many competitors battling on feature parity. He believed the real value for organizations was in security, where application-specific mistakes—like improper RAG access controls or agent tool vulnerabilities—cause major business risks. 'You can't fix stupid,' he notes; foundation models can't prevent application-layer errors, making pre-deployment security testing the higher-impact opportunity. [5:55](https://addtry.com/9ca4c648-b223-4d97-a551-d60a1e0ec614?t=355000) ### Which companies are using Promptfoo for AI security testing? Over 10% of Fortune 500 companies use Promptfoo, including major household-name brands. Ian says these large enterprises face board-level pressure to deploy AI, but their security teams lack tools to manage risk. Unlike startups that ship quickly, Fortune 500s have the most to lose and are increasingly making systematic pre-deployment red teaming a hard requirement for releasing AI features. [25:12](https://addtry.com/9ca4c648-b223-4d97-a551-d60a1e0ec614?t=1512000) ## Key moments - **[0:00] Intro** - **[0:31] Origins** - [0:40] Ian Webster started building Promptfoo as an open-source tool while leading Discord's AI chatbot Clyde. - [1:45] Clyde was Discord’s AI chatbot designed as a social bot to encourage interactions in channels. - **[3:41] Promptfoo Defined** - [4:43] Promptfoo’s evaluation approach was inspired by Google's side-by-side tables for comparing search algorithm changes. - **[5:40] The Pivot** - [5:55] Ian pivoted Promptfoo from evals to security because evals are a commodity; the real value is in application-specific risk. - [8:17] "You can't fix stupid" — Ian Webster says even the best foundation models can't prevent application-layer mistakes. - **[9:49] Demo** - [11:22] Promptfoo uses AI-vs-AI to generate tailored attacks based on each application's business context. - [12:46] Q: Does Promptfoo automatically discover application context? Ian says manual entry is preferred for accuracy. - [14:35] Promptfoo’s plugins generate attack objectives tailored to the app's purpose, reducing false positives. - [15:06] Promptfoo covers OWASP LLM Top 10, NIST, and EU AI Act compliance through over 100 risk plugins. - [18:21] The red teaming loop runs attacker and judge LLMs up to 1000 times per objective to refine attacks. - [18:51] Q: How does Promptfoo stay current with AI security research? Ian uses Google Scholar alerts and a 12-person team. - **[22:28] Remediation** - [22:28] Remediation for AI vulnerabilities is limited: harden prompts, tune guardrails, or fix code directly. - **[23:48] Traction** - [24:43] Over 10% of Fortune 500 companies use Promptfoo; large enterprises have board-level AI initiatives driving adoption. - [25:12] Open source is crucial for Promptfoo to build trust in a space full of snake oil, says Ian Webster. - **[26:45] OSS to Enterprise** - [26:45] Q: What's the difference between open-source and enterprise Promptfoo? Enterprise adds triage, monitoring, and validation. - **[27:56] Landscape** - [31:18] Companies realized guardrails alone can't eliminate risk; pre-deployment testing is now required to measure AI risk. - [33:29] Ian predicts that agents with 30-minute runtimes will break current black-box AI security testing approaches. - **[34:11] Future Plans** - [34:36] The future of AI security lies in pre-deployment testing integrated into IDEs and CI/CD, closer to developers. - **[35:09] MCP Security** - [35:09] Most enterprises use MCP as glorified API wrappers, reducing MCP security to basic API security. - [39:20] Corporations struggle with local MCP servers; employees running NPX commands risk unvetted tool access. - **[41:42] Takeaways** - [41:42] Ian: Measure AI risk early in CI/CD to prevent production issues, as risk management becomes a hard requirement. ## Speakers - **Alessio** (host) - **Swyx** (host) - **Ian Webster** (guest) ## Topics Security ## Mentioned Anthropic (company), Character AI (company), Discord (company), Dropzone (company), GitHub (company), OpenAI (company), Proofpoint (company), Push Security (company), Sublime Security (company), Azure Content Filter (product), Bedrock Guardrails (product), Clyde (product), EU AI Act (product), GPT-3.5 (product), GPT-4 (product), MCP (product), NIST (product), OWASP (product), Promptfoo (product) ## Transcript ### Intro **Alessio** [0:04] Hey everyone, welcome to the Latent Space Podcast. This is Alessio, founder of Kernel Labs, and I'm joined by Swyx, editor of Latent Space. **Swyx** [0:12] Hello, hello. And today we have Ian Webster from Promptfoo with us. Welcome. **Ian Webster** [0:16] Hey guys, good to be here. Definitely excited. **Swyx** [0:19] Yeah, I first came across you when you, like, submitted a proposal for AI Worlds Fair last year. Uh, and then, and then shortly after you started, uh, you know, spinning out Promptfoo into, like, its own, its own company. You just announced your Series A, uh, 18 million with Insight Partners and investors in Horowitz. What's the journey been like? Yeah. Tell, tell us about your journey. ### Origins **Ian Webster** [0:40] It was kinda crazy. Like I, I have always been an engineer because I, I love building things and I've, I've had successful open source projects in the past, but th- this is by far and, in a way, the one that has, has gone the farthest, right? So I, I started Promptfoo open source while I was at Discord. Um, and at, at Discord, um, I mean, I joined Discord in, in 2020 and, and started the developer platform organization essentially. Um, and then when AI got, got hot, I switched over to, um, kinda leading some of, uh, Discord's AI efforts. So it was really through, like the, the pain of, of having to ship AI to 200 million teenagers that, that I realized, like there, there's definitely something here, both in terms of evaluation and then also, like risk management. **Swyx** [1:32] Yeah, yeah, just as a side, like Latent Space obviously runs on Discord as well. Um, and uh, you know, obviously we're very interested in AI. Clyde has been an interesting journey for Discord. I don't know if you wanna comment on like what you, what you did with Clyde or like what happened. **Ian Webster** [1:45] Yeah, I mean, I-- So I, I, I led Clyde basically. Um, and I, I, I guess for, for those who, who don't know or haven't followed, Clyde was like an AI chatbot on Discord, um, kind of what you expected. Um, and it, it was in the heyday of like GPT-3.5, right? So everyth- everyone was like, "Let's, let's do a chatbot." Um, for Discord, in my opinion, like it actually makes sense, right? Because chatbots and Discord. The interesting thing about Clyde was that it wasn't really meant to be a necessarily a, a helpful bot, but instead more of a social bot, like a, like a bot that would just like be in your channel and, and, you know, help people interact and that kind of thing. So yeah, we, we built out that concept that I feel like since then has kind of been carried forward by, by character AI and, and like a, and kind of like a handful of companies in that vein. **Swyx** [2:40] xAI. Yeah. **Ian Webster** [2:41] Makes sense. Yeah. **Swyx** [2:42] Uh, Shapes. Shapes was big. Discord has been cracking down on Shapes. You know, you don't have to comment on that, but, uh... Anyway, so like, uh, was that the main AI effort? Were there other AI efforts that maybe you don't know about? What was your experience that you're drawing on, drawing from to make Promptfoo? **Ian Webster** [2:56] Yeah. There, there were others. I think the ones that are probably most visible are like AI summaries, which appear on certain, um, like, like large channel, graphic channels, as well as there was, there was some stuff on like the customer support side, I think. And I'm-- Like, by the way, I'm, I'm long gone from, from Discord, so I, I, I'm probably free to... Like I don't know anything that, that shouldn't, that shouldn't be, uh, said publicly or anything like that, and I, I don't have any context on the Shape stuff. **Swyx** [3:26] Yeah, cool. Um, so we can dive right into Promptfoo. Uh, it started as like a, like a, kind of like a eval testing toolkit, uh, has, has broadened out into the broader sort of security problem. Maybe let's get like a, like we'll start from like what it is today, and then you can, you can fill in any history you need. In a crisp way, what is Promptfoo? ### Promptfoo Defined **Ian Webster** [3:43] Yeah. So Promptfoo is a way to find and fix issues or risks in your AI application. So we have these days primarily a security focus, or at least that, that's our go-to-market, but we're also just a very popular open source evals tool. Um, and as, as you mentioned, like we originally started as an evals tool, and I can, I can fill in a little bit of context there. The inspiration from Promptfoo came from basically we, we shipped this thing, like we, we yellowed this thing to, to like a couple million users, and then we quickly found, especially 'cause this was like before GPT-4 was, was economical, we quickly found that you would, you would pull on one end and, and something drastic would change on, on the other. Um, and I spent so long trying to, trying to get it to, to basically talk casually instead of, instead of like an LLM. So it, it quickly became apparent I needed to do something systematic here. I needed something for the team to do that. So that's, that's why I started building Promptfoo. The inspiration came from actually like 13 years ago in my Google days, I used to work on improving search. And when you had a change to propose to the algorithm or whatever, the way that it would be evaluated is it would go to a bunch of evaluators, and it would come back as like this really ugly table with column A and column B, and you compare the two. We, we called them side-by-sides. So all I wanted every day was just like a side-by-side for what we were working on at Discord. And that, uh, you know, that was even before like the term evals was, was coined or, or popular. Like, I remember having to explain to people what Promptfoo was, even though I'm sure like the, the ML people have been doing evals forever. But I guess the, the point is I just wanted like a super easy side-by-side interface, and that's how it started. Um, and it just kind of grew and evolved from there. ### The Pivot **Alessio** [5:40] What were the initial maybe forks? You know, you could've gone a lot of different ways, right? Now you ended up with security as the main go-to-market. Are there other branches of the story that maybe you didn't go through that you found interesting or any other kinda like idea maze exploration stories? **Ian Webster** [5:55] The obvious fork here was do I lean into being an eval tool and- Maybe an observability tool, or do I find something else to do? And I, I think, like, the, there, there was a bunch that went into that decision, but in my opinion, the main thing that, that really drove it for me was that I think evals are, are table stakes. I think that they're a commodity, and everyone should be doing them. And yes, there are companies that are doing great in the eval space, but to me it just seemed like a bloodbath. You know, like, we would just be ... Had a great start with Promptfoo, and we would've just been fighting on, like, feature parity, tit for tat until the end of time, going enterprise by enterprise, and just, like, that's a, that's a long slog. Not that what I chose isn't a slog, but you know, it sound- it sounded better to me. **Swyx** [6:44] It's a different slog. **Ian Webster** [6:45] Yeah, a different slog. And I thought to myself, what is the best, like, what is the, what do, what do I see people doing with evals that has the most value to the organization? And the, the security safety component, I think, was, was, like, a very obvious choice. **Swyx** [7:02] And then there's kind of the question in security of, like, security because of the model, security because of your prompt and kinda, like, the way you're maybe, you know, executing prompts and things like that. How did you think about that? I think initially there was, like, this whole, like, oh, toxicity and, like, all these different kinda, like, base evals that people run, but obviously the model providers themselves ended up mostly fixing the problem in the models. How did you think about what is kinda something that is worth for you to build versus, hey, it's just gonna be fixed upstream? **Ian Webster** [7:33] Yeah. So broadly speaking, there, there are two buckets, right? There, there are, like, the foundation risks, which is toxicity and stuff like that, and then there are the application risks, which are all the ways that you can shoot yourself in the foot, like, by, by, for example, not adding proper re- access control to, to your RAG. So, like, it, it can access documents, um, that you shouldn't have access to, or giving agent tools or API capabilities that, that, again, don't have proper access controls or, you know, can do un- undesirable things. I always tell people, like, you know, OpenAI, Anthropic, they have, they have teams of geniuses who are, who are working on making sure that models aren't toxic and that kind of thing, but they can't ... Well, the, the, the flippant way to say it is, like, you can't fix stupid. I don't, I don't think, like, the, the people who are necessarily testing this are, are, are stupid, but, like, everyone makes mistakes, and that's what a lot of security is, is built around, or a lot of the s- the cybersecurity industry is built around. And there have been many, like, pretty high-profile instances of this. So, so yeah, I'm, I'm a lot more interested in what happens at the interface between the application and the model and what can go wrong there. And I think definitely in the fullness of time, that's where this domain is most interesting by far. There's, there's a kind of middle ground as well, which is there's, there's, like, a slight difference in incentives between the, the application developer and, and these foundation labs, which is that OpenAI, Anthropic, everyone else, they're all building models that are, like, maximally helpful, and in actually most cases in a corporate environment, you don't want that to be maxim- you, you don't want the model to be, like, helpful in every way possible. So in between toxicity and this application risk or, like, like, hard te- hard technical security risks, there are softer but still relevant risks. Like, you know, healthcare companies don't want their bots to recommend off-label drug use, or people don't ... I guess a, a boring example is, like, people don't want their bots recommending competitors. Stuff, stuff like that where there actually is a, a delta that you have to test. **Swyx** [9:49] Yeah. Perhaps we, we ... I don't know if you wanna ... It's time to go for a demo. We tested the, the screen share capabilities. Is there any place that you recommend starting for trying stuff out? ### Demo **Ian Webster** [9:59] Yeah. Um, I definitely can do that. I'd be happy to. So, um, I have a ... I'll kinda walk you through how, how Promptfoo works, and tell me, tell me if this is, like, too in the weeds. Um- **Swyx** [10:18] Nothing's too in the weeds. **Ian Webster** [10:20] Okay. Let's- **Swyx** [10:20] Like, yeah. E- e- especially for, like, a developer tool for, like, that crosses AI and security, like, we are the perfect podcast for this. **Ian Webster** [10:29] Tell me if this is too not in the weeds then. But basically, like, like, I'll, I'll walk through what Promptfoo does under the hood and as, as well as, like, what the, the, the setup looks like and what you get out of it. But basically, when you use Promptfoo, you, you bring two things. Obviously, the target application or model, as well as information about the application use case and, and that kind of thing. So the target configuration is very straightforward. Like, most people are hitting an HTTP endpoint, but, um, but it can also hook directly into, like, your Python, your JavaScript, whatever, and we have support for 50 different inference providers. So that, that's, like, the, the, the boring part. I guess the, the more interesting and, like, much more impactful part is information that you provide that's relevant to the purpose of the application and its business context and so forth. So we, we ask people to ... When, like, when I, when I talk with people about how to use Promptfoo, we ask you to fill out as much of this as possible because this is all gonna have pretty substantial downstream effects on everything else that, that happens. So you tell them, you ... Or you tell, you, you enter in here, like, what's the purpose of your application, et cetera, what should it and should it not have access to, is there any additional business context, is there, like, a type of, of user or persona that the attacker should imitate, and so forth. And maybe before I, I go further, like, I, I probably should've backed up a second and just explained high level. The way that Promptfoo works is by using- AI versus AI in the sense that we generate attacks that are tailored toward your application, your business context, in order to find things that you are worried about or in order to find potential risks. So this is, like, distinct from having a static vulnerability database, for example, or just like a list of 1,000 canned prompt injections that, that we send through. Everything is kind of generated on the fly, and that's what makes it, like, useful and, and, and, and relevant to application security teams. Um, so any, any questions so far? Like, is this, is this the, the right level of detail for you guys? **Swyx** [12:46] Yeah. **Alessio** [12:46] Yeah, keep going. **Swyx** [12:47] Um, one question, how does the Discover button work? Like, when you were running through this, kinda makes sense that if I give you the website, you should be able to infer most of these things. Is that something that people use? Are you happy with how good the models are at kinda like pre- prefilling these things, or is there still a lot of, a lot of context that the users go in and input manually? **Ian Webster** [13:07] Yeah. Honestly, I think I always encourage people to, to fill it in manually whenever they can. And the reason for that is the people who built the system, as well as the security guys, they have a whole lot more context than what could be discovered on the fly through the front door. So if you're doing a, a, like, a black box test, like you just give it an HTTP endpoint, the most that we can do from a discovery perspective is just, like, talk directly to the application. Which, you know, may- maybe we can enumerate some functionality or guess at the underlying tools and so forth, but the, the team that actually did the implementation is gonna know a lot better. So I think that part is, is pretty important. There, there's some, some nuance here. So if your application has, has tracing, like, if it can export to an OpenTelemetry sync, then we can learn a lot more about the application. I think the other thing is if, if you are testing an MCP client or server, we can do the, the, like, list tools on, on MCP to get, to get more information about that as well. But I would say the majority of people approach this as, as, like, a black box test, which, which is actually not the best approach in, in my opinion. I think especially where, where things are headed, like with more complex RAGs and, and agents and so forth, you're going to have to have some type of observability or, like, internal tracing in order to have to do meaningful automated red teaming. Yeah, I guess with, with that said, you know, so the, the next step is to choose what we call plug-ins. So plug-ins are basically different risk areas. We have, we have over 100 of them at this point. They're built on top of either uncensored models or, for some proportion of them, off-the-shelf models. Really depends on, on which risk areas you're, you're focused on. And these plug-ins produce application-specific attack objectives. So we have a bunch of presets out of the box. On the security side, we see a lot of people caring about the OWASP LLM top 10. And then in the US, to a lesser extent, we see people caring about NIST, and of course, the EU people, um, are, are interested in the EU AI Act. And then the areas that you can cover are, are broken down into four sections. There's, like, straight up technical security risks. There's compliance and legal, so I guess things that you can get in, in trouble for potentially. And then softer trust and safety, toxicity issues, as well as brand or reputational risks like hallucination. And we also have a, a handful of domain-specific risks in... I'm, I'm sharing the open source right now. In the open source, we have them specifically for, for finance and, and healthcare. So that's, that's the, the, the gist of it. These attack objectives, and I can, I can show you what an example looks like. Um- **Swyx** [15:54] J- just 'cause you have the, uh, the, this on, that on screen, uh, you should, you should also probably share that you're a member of the OWASP, uh, committee. **Ian Webster** [16:01] Yeah. Yeah, I, I... Yeah, there, there, there are a handful of, um, of, like, OWASP groups or committees that are, that are focused on certain issues, and I, I was an author on the, the OWASP Gen AI Red Teaming guide, for what it's worth. Um, I think it's actually a pretty decent guide. The problem with these, these standards is that things are changing, you know, daily, but, like, a lot, a lot of the guidance is, is, is incomplete in the, in the three to six-month timeframe. Um, so yeah, OWASP came out with their first LLM top 10, uh, last year. They had to update it in January. I imagine it's gonna be updated again. **Swyx** [16:40] Yeah. **Ian Webster** [16:40] So one of the issues is that the space is moving pretty fast. Yeah. So and the way that plug-ins work is that everything is based on the, the application details that you've entered. So if, if I put in, like, customer... Or I can, I can put in whatever you guys come up with, but I can do, like, customer support chatbot for global banking, or if you have any suggestions, I can do that too. And then I can roll the dice here. I'm always nervous because this is, this is, like, on the fly, but you know, it's already talking about, like, pins and transaction data, et cetera. Um, so this is, uh, this is just an, like, a one-off example, right? We normally would put in a lot more information here, but you get, you get the idea. Like, the, the, the attacks that these plug-in or the objectives that th- that these plug-ins generate are very much tailored toward the use case. And then from there, these get fed into what we call strategies, which are basically different attack methods. Um, um, so most of these strategies come out of published ML research, and then we tweak them, or we build on top of them in order to make them, like, actually effective on, on application use cases 'cause most of the research is done focused on foundation models. Um, so we, we kind of modify a lot of these optimization or search strategies to bring in that application or that, that business context and use it accordingly. And This is an oversimplification, but typically there's an attacker LLM that communicates directly with a target application, and then a judge which looks at the outputs and is like, "Well, that didn't-- like, that got blocked by guardrails for, for this reason, so let's tweak it." And this loop can run a hundred or a thousand times per, per attack objective. So that's, uh, that's, that's kind of what the configuration looks like in a, in a nutshell. Any questions here? Making sense so far? **Alessio** [18:34] Yeah, that makes sense. One question on how you keep track of all the papers and, like, research and whatnot. Have you figured out some easy way to do it? Or how do you... Like, you know, given that you're spending some time to make it better for your use case, like, how do you figure out what is worth adding, what is not worth it? Like, do you have a team internally that kinda takes care of that? **Ian Webster** [18:54] Yeah. Um, it's, it's basically a lot of Google Scholar alerts. That's, that's, like, the main thing there. And the team here is still really small. We're just, like, 12 people. So I wouldn't say I have a team internally, but, like, we all keep an eye on it. And there's, there's a lot of research coming out in this space. There are probably five to 10 papers per week, so it's, it's pretty noisy. And it's become very easy to just, like, like, publish, quote unquote, something around just, like, a random jailbreak that, that you found. So we, we quick- we quickly stopped incorporating just, like, any jailbreak that, that made it onto-- Yeah, that, that, that someone decided to, to kind of, like, dress up. But if, like, a new fundamental technique or approach comes along, we'll take it, we'll try to reproduce it, and then we will eval it against, like, actual real-world application use cases. **Alessio** [19:43] Cool. All right, so now I set up my whole app. You have decided which ones are the things to care about, and then from here you go into the run? **Ian Webster** [19:52] Yeah. From, from here you run it. So I, I guess I configured this against a dummy endpoint. Maybe I, I won't do a live run, but I do, I do have a couple. I don't-- Yeah, let's, let's just see what happens if we click Run now. I guess I did fill out all the information. So it's, it's gonna basically go through this, this process where we generate and then we eval. The way to think about this, I guess for, for the ML audience, is we're just a synthetic data generator for, for your, for your eval test set. Um, that's, that's actually what, what, like, the Promptfoo security red teaming solution is. And then, like, long story short, you know, this, this will take, depending on the size of the run, as well as your, your, like, throughput on your application, maybe it takes 20 minutes, and then you wind up with a report that looks like this. So in this particular case, we're red teaming a customer support chatbot for a hypothetical bank. We had 27,000 conversations with it. And you wind up with basically this report card across, um, a bunch of different areas: security, compliance, trust and safety, and brand. You can click into these and see some inputs that were flagged. So I think this chatbot was built on top of Foro or something like that. So there are some, uh, kind of, like, jailbreak-y sounding things that look like this. And I guess I would point out that under the-- there's always an application-specific ask here. Um, and then we also, depending on the attacks that you've chosen, we also have, uh, kind of like longer form attacks where the attacker starts out with, like, an innocent question and then slowly over 10, 20 back-and-forths, um, leads the LM down a path toward where it is more vulnerable or more, more willing to, um, um, like give you, give you what you want. So in this case, we-- this, this was built on top of a, a RAG with mock medical data, and we were able to extract some, some PHI. So this is what it looks like in, in a nutshell. I will also say, like, under the hood there, it's all in eval, right? So I can show you the, the really ugly eval view that I, that I dreamed of back at, back at Discord that I, that I wished that I had, if it loads. But, but yeah, you know, you can, you can see all the, the results here, and this is kind of the less, uh-- the, the report view is, is like-- the report card is like eye candy, but this is the operational view, the thing that people, like, actually use to, to understand what is going on and, and how it's, how it's working. Hope- hopefully that, that's, that's helpful. That's what Promptfoo is in a nutshell. **Alessio** [22:28] Yeah. What do people do from here? So you're saying basically my app sucks. There's like a lot of, a lot of security issues. What, what's the, what's the next step? ### Remediation **Ian Webster** [22:38] Yeah. So there, there are-- when you have a finding there, there are certain things that you can do, obviously, depending on the finding. But the, the list is honestly not that long in, in gen AI. If it, if it's a foundation issue, you can harden the prompt, you can tune your guardrails or, or like implement a guardrail. And we, we have a guardrail solution that is trained by these red team findings. But honestly, like, no matter the guardrail, you'll probably have to do some tuning. And then if it, if it's like an API or access control issue, then you just have to fix your shit. Like there's, there's no, there's no amount of prompt hardening or whatever that's, that's gonna be acceptable in, in that sort of situation. So in our commercial version, like 'cause a lot of large companies are like, "Okay, I have all these problems. What do I do now?" In our commercial version, we have a, like, remediation path. So we tell you like, "Okay, for this one, here's how you update your prompt," or, "If you're using such and such guardrail, like here, here's what we recommend tuning it with." And I, I know you can also, like, you can, you can fine-tune models and that kind of thing, but for, for, for most of the teams that, like, at big companies right now, your options for remediation are pretty limited. **Swyx** [23:48] It's like what, what type of companies are, you know, resonating the most? You know, like, I think like, um, it depends, obviously, the scale of the app that you're running against, and also, like, the resources that they have to, to, to dedicate it to this. ### Traction **Swyx** [24:04] And I think, like, security sales is, like, a very hard sale. It's something that historically I haven't been involved in. And I always think like, to some extent, like it's kind of like fear-based marketing because- I, I guess, like, you know, there's, there's some legal liability, but really it's just like, you know, there's the never-ending list of issues, right? Like, like, uh, I'll never be done fixing security bugs 'cause there's, there's so much. **Ian Webster** [24:24] Yeah. Security's in a strange place right now because you have all this top-down pressure to, to, like, AI, AI everything, right? Even at the world's largest companies, there's some board-level initiative that's like, "Do more AI." And a lot of security teams are, are, like, flying blind or they're scrambling to, to catch up. We are actually seeing the most commercial traction from basically the, like, the, the world's largest companies, like, you know, Fort- Fortune 500 types. And I think the reason for that is that those are the guys with the most to lose. If you're a startup building on AI, like, you're, y- you're gonna ship it no matter what, right? Like, it... You're, you're probably not gonna take time to, like, to red team and to come up with a guardrail solution that, like, has a feedback loop from the red teaming, et cetera, et cetera. So I've noticed this, this resonates mostly with, like... w- with, like, household name brands, essentially. We have a bit over 10% of the Fortune 500 that, that use Promptfoo right now. Um, I expect that number is gonna continue to grow pretty substantially because, um, this is basically an initiative at every single one of those companies. And just to, to, to add on, I think you're right that, like, a lot of cybersecurity stuff traditionally is, is a fear-based sell. I, I try to be a little careful with that because security people, by nature, they tend to be very skeptical people and very skeptical of people trying to sell them things, just because cybersecurity is, like, so built on top of, or, like, there's so many vendors in cyber, and there's so much snake oil and smoke and mirrors in cybersecurity and in AI. When you put them together, it... I don't think selling on fear alone works very well. That's why I think open source has, has been really good for us because there's so many people selling stuff that maybe doesn't really exist or doesn't really work in this space. Whereas you can, you can NPM install Promptfoo right now and, and, like, start doing this stuff, and it's, it's pretty clear we have something that's real and something that's effective. So that's been my number one way to, to, like, get in the door or get in the good graces of folks who are doing work in this space. **Alessio** [26:45] What's the, um, kinda like open source to enterprise jump mostly? Like, do people get most of the features in, in open source, and then it's more about the cloud management, or how does that work? ### OSS to Enterprise **Ian Webster** [26:56] Yeah. So everything I showed you is in, in open source. The, the way that I think about it is that if you're at a big company, finding the issue is only, like, 20% of the challenge, and then the rest is you have to triage it, you have to, like, hook it into your task tracker and your SIM and, like, all these other systems. You have to monitor it, you have to remediate it, you have to validate the remediation. So there's a very long list of things that are required to, to complete the full vulnerability management, um, life cycle, and that's what we focus on on the enterprise side. Um, so I, I definitely... You know, we have all the bell, the enterprise readiness bells and whistles like the, um, compliance certifications and SSO and RBAC and professional services, that, that type of stuff. Um, I, I didn't wanna build, um, a, a, a company on, on top of just, like, support and services pretty much. So focused on, on the vulnerability cycle. **Swyx** [27:56] Nasio, how do you think about the, the, the landscape of tools you see? Like, how does Promptfoo kind of fit in the overall trends that, uh, customers are wanting? ### Landscape **Alessio** [28:05] Yeah. It, it's interesting because this is kinda like a new area altogether. I think, like, a lot of what we're seeing get a lot of adoption is, like, using AI to change the shape of old security tools. So we have one company, Sublime Security, that does email security, and it used to be that, you know, Proofpoint and all these tools have, like, these black box models that do detection for you. And what Sublime did is they have a DSL where you can write custom rules, and then you can use, because it's code, the AI can help you write rules in real time. So whenever a new vulnerability is out, you can, like, in real time write a rule for it in your system. Same thing is happening with Push Security, which is like a identity access security company. So we're seeing less almost of, like, securing the AI itself. I think the, the, that's still, like, an area that people are trying to figure out what to deploy, you know? So it's almost like trailing. I think MCP security is one space where we're seeing a lot of things because I think it, it kinda makes sense to people, which is like, okay, I'm just making sure that this connector that used to be an API that is now an MCP server is secure the right way. I think that makes a lot of sense. Threat hunting is another space where AI's having a lot of success, like how to use AI to, like, threat hunt. AI SOC is another one. We have Dropzone that it's growing really fast, how to use AI to, like, get through all the notifications. Because this is similar in a way to what you see in Promptfoo, which is, like, there's, like, a lot of things that can go wrong, but then there's a limited amount of ways to fix them. So AI is actually pretty good at, like, matching the issues to the fixes and help you navigate through them. So but yeah, in, in this space, we haven't seen as much, and I think that's why, you know, Ian is getting so much adoption, which is, like, a lot of, a lot of the tools that we've seen in the first wave, like, you know, I'm not gonna name some of these companies, but they were maybe like, "Hey, you know, we'll secure AI," they raise 30, 40, 50 million, and then you kinda get there, and it's like, how exactly do you secure the AI, you know? And it's more top-down versus I think Ian's approach of being bottom-up is like every, the security team and the engineering team can work together to fix this because you have specific issues, and you have ways to implement that in the code, versus I think the first approach was like, "Hey, we'll be top-down, maybe we'll be- A proxy on top of your LLM. So instead of like looking at your code, we'll kind of take all the prompts that come to you and like take a first look at them. I think that obviously didn't work for clear reasons, but I'm curious seeing what, what you're seeing of like what tools you're ripping out, you know, like what people tried before that didn't work. Obviously, you don't have to name specifics unless you want to. We're always hoping for hot takes. **Ian Webster** [30:44] Security for AI is pretty greenfield, so most of the people who are bringing Promptfoo are not replacing anything. It's just like that, that, that's part of what makes this such like a... It's the reason why investors are excited about this opportunity, right? Because it's like a whole new category in cybersecurity, which you don't really like, like that doesn't come around very often. So it's, it's a competitive space, but not so much because of incumbents, but actually just because there are a bunch of other startups in, in, in this area, 'cause like people have felt the pain and, and recognized the opportunity. I think, I guess like in, in the first wave, I agree, everyone was, everyone is fo- was focused on guardrails because it, it fit the mold of what people were, were used to, kind of like a WAF style thing. There are a lot of really big security companies that made their fortunes on having like a network or point of presence and, and that kind of thing. But like people are realizing that there's no such thing as a perfect guardrail, and there are so many, so, so many projects that are blocked at huge companies because it's not, it's not like palatable to say, "Okay, we know this has serious issues, but like, fuck it, we'll do it live. The guardrail will catch it, and we'll just pray to, to, I don't know, the guardrail gods that, that they'll, they'll do it." So they're... So like this year, basically, people have realized that, "Oh, we need something pre-deployment that can actually, that can actually measure the risk and, and allow us to manage that risk." And I, I think that's where startups can, can get more involved. I know that there, there were a lot of guardrail startups, or there's, there still are, I guess. But building a business on guardrails really scares me because there are so many incumbents that can come in and, and eat your lunch. And it's like, it's not actually that hard to, uh, to build a guardrail. I don't know if I'm gonna make people angry by saying that, um, maybe some, some of my competitors, but like, I think guardrails are, are a commodity, um, as evidenced by the fact that like ninety-five percent of the people that we work with are just using whatever guardrail comes from their inference provider, like Bedr- Bedrock Guardrails or Azure Content Filter or whatever. So the future is really like not in runtime, but earlier in the software development life cycle, ideally in CI/CD or before that. Where Promptfoo sits right now is developers can run it as a CLI, or you can bake it into CI/CD as a sort of regression test. But I actually think, like, so, so I think that's our, that's where we're doing very well today. I kind of think that that is gonna stop working or it will stop being enough, as I mentioned earlier, because of the complexity of, of like agents and some RAGs. So like what we're seeing now is like we've encountered some agents that just like kick off and run for, for half an hour, right? And like how, how can you do a good job testing and validating its security if all you're doing is talking through the, the front door, right? Like in the chat input and looking at the output. So that's maybe a long-winded answer, but I don't even think Promptfoo as it exists today is gonna be adequate in the future, which is obviously what we're working on internally is like what does, what does it look like to, to actually do security for agents? And yeah, we, we have a bunch of ideas there, but the sh- the very short way to summarize it is like that security tooling needs to be closer to developers as opposed to kind of after the fact. ### Future Plans **Swyx** [34:11] Yeah, I feel like a, a few people have exploited this observation in the pre-GenAI era, and I'm curious if, if there's sort of any role models that like come to mind that like, uh... Yeah, I, I'm, I'm definitely learning about the space as we, as we, as we talk about this. But yeah, yeah, I mean, like basically, do you, do you, do you want to see this as an IDE plugin or like, you know, like how, how, how extreme close to developer do you-- can you get? **Ian Webster** [34:36] Yeah. Uh, so I, I, I wanna do an IDE plugin. We're, we're experimenting with, with, um, some of that. And then also something that, that's more code aware. Like, I want you to be able to, to hook it up to, to GitHub and, and get a little more insight that way. So not exactly like static code scanning, but I mean, Alessio asked about the, um, like that discovery process right now. I think that discovery can go a lot deeper. It should be code aware, um, and, you know, things like that. We-- Yeah, that's the sort of stuff that we're working on. ### MCP Security **Alessio** [35:09] What else? So you had the-- I saw the MCP proxy thing. Uh, what's interesting there? I would say the questions that people have today is context exfiltration through MCPs taking bad actions, so destroying things accidentally. What are like the different attack vectors maybe that you see, the failure cases? What are people trying? **Ian Webster** [35:30] Yeah. So, so Promptfoo has offensive testing capabilities around, uh, basically everything that you outlined, like the, like the different threat scenarios. So there's scenario number one... Well, here, let me just show you. I don't, I don't think this is gonna, uh, I don't think this is gonna shock you or anyone who sees this, but this is basically how, how we think about it. So we see a lot of people obviously just like building internal use cases on top of MCP, where you have a trusted first-party client and a server. Um, and that's pretty straightforward, and we just wanna make sure that their MCP server, like they, they didn't naively wrap an API and it can leak an API key or pitfalls like that. I think definitely becomes more interesting when you're in an untrusted environment. So we have dealt with a, a handful of cases where obviously you, you have like the, the rogue or evil MCP server that can tool inject or kind of man-in-the-middle, uh, tool calls in a, in a sense. Uh, and that's something that I think is, is like a pretty fundamental ... problem for, for some of the use cases that people are trying to develop MCP for. Um, but we, like, we have our own, uh, or Pr- Prompt- Promptfoo can, can simulate that rogue server. So you just kind of point it at your server and we test that. Um, and then Promptfoo can also talk, um, MCP directly and just send JSON into the MCP server, which in my opinion is, like, I don't know how useful of an exercise it ... Or, like, it, it c- it basically reduces to, to API fuzzing if you just talk to- **Alessio** [37:12] Yeah, MCP security is API security, right? **Ian Webster** [37:15] Yeah. So that, that's what, you know, I, I, I struggle with this sometimes talking with people on the security side because, like, a lot of ... It's, it's kind of a lot of what they know al- already, and I, I know that MCP can be much more, but the way that a lot of corporates are implementing MCP is just like a glorified API wrapper. Not, not, like, super innovative, um, just kind of checking the box. So we have these offensive testing capabilities, and then Alessio, as you, as you mentioned, we also have an MCP proxy which does a couple things. It basically, it provides some observability, and it, it is an MCP server that, that's essentially a forward proxy, and then it, it also, um, uh, containerizes MCP servers so you can control for, for a given server what it has access to on the file system or, like, which network hosts can it, can it speak to and that kind of thing. And that's our, you know, like our ... It's, it's us branching onto the, the d- defensive side as well, I guess. **Alessio** [38:17] Yeah. **Ian Webster** [38:17] Yeah. **Alessio** [38:17] I think there's not a lot of usage of any of the other primitives outside of tools in, in servers. I've been building this MCP client called KernelGem, and we support the registry and blah, blah, blah, and most of the servers are just a couple tool calls. They don't really do anything else. So I, I'm curious, like, you know, I, I think resources is, like, another attack vector that people are gonna be use more of, you know, which I think people don't do today. Yeah, because that happens, like, you know, the resource gets loaded before the inference gets run, so to speak, so it just gets moved in whatever runtime you have before it gets done. I'm curious how that's, that's gonna be secure more. Sampling is another one, so the other lamps can, like, take the, like, the server can load data and then run inference before it returns to your main inference flow, and that's another thing where, like, the user doesn't even see really what happens there. You just get the final result. So yeah, it's a curious space to, to be in. Yeah, right now it's like a ABI wrapper, wrapper. **Ian Webster** [39:20] I'll ... Yeah, I, I will also add, like, one thing I've observed in the real world is, like, the thing that drives security people crazy is that, um, is, is, is local MCP specifically, right? So in, in order to, to, to even do a, um, set up an MCP proxy and have, like, a, like, centr- centralized observability and control that, um, you need to prevent people from, um, from just, like, like, NPX running their, their own stuff locally, which I know, like, to us it's like, "Oh my God, why would, why would you do that?" Because that's, like, one of the joys about the MCP ecosystem right now is you can throw in whatever you want. But if you imagine from, if you're, like, a corporation with 100,000 employees, that's probably not something that would be okay. So they're trying to figure out how to, how to do that first, um, let alone whether an MCP proxy or, like, how to centralize that, whether, whether that makes sense. **Alessio** [40:23] Yeah. Are, are you doing the, the hosting for the MPX one? That's one thing that I've, I've done this with, um, E2B instead. So we have, um, MPX servers too, like this one, and then we just spin up a E2B sandbox and run it in there, and then it's, like, paired per user. Each user gets a sandbox, and then you can kind of mix and match. You know, this is like a remote one, and then you can match them together. Most people that I talk to, especially in lar- larger enterprises, are like, "I don't really know if I want you to do that. You know, I maybe want to own, like, the runtime," so I'm curious what you hear on the proxy side too. **Ian Webster** [41:02] Yeah. For, for the really big companies, something like this would probably have to be on-prem, at least, at least in my experience, just because there's ... You can get caught up in security reviews and stuff like that for, for, like, quarters on, on end. So yeah, a lot of people are not ... On-prem is its own special type of headache. But right now it's still early days. Like, that's kind of the first thing that I'm seeing people look for is something that they can control, like, within their own environment. **Alessio** [41:32] Yep. What else? You have, um, LM security database. You got a lot of cool things that maybe are not the main, the main course, so to speak. Uh, what, what else you wanna highlight? ### Takeaways **Ian Webster** [41:42] Probably nothing else I wanna highlight. I think the, the red teaming is the main highlight, and I guess I would, if I could, like, give people a, a takeaway, it's that if, if you're worried about controlling risk in your generative AI applications or features, then it makes sense to be able to measure that risk early on before deployment, basically as early as possible. So if you can run tests after you have developed the product but before you promote it to production, that's great. But ideally, you, you bring it into your CICD and have that sort of assurance, like, on the PR itself or otherwise earlier in the process. That's just kind of my, my two cents or, like, the, this is, this is the horse that I'm, I'm beating every day. But it, it seems like at a lot of big companies, this, this is becoming a hard requirement for gen AI. **Alessio** [42:36] Well, awesome. Uh, well, people should get in touch. I assume you're also hiring. Any other, uh, final calls to action? **Ian Webster** [42:40] Yeah, um, that's a good point. We, we are hiring basically a- across the board. So if, if, uh, securing AI or making AI behave properly is, uh, is something of in- or behave improperly, I guess, is technically what, what we do on the red teaming side. If any of that is interesting to you, would, would love to talk, so please get in touch. **Alessio** [43:01] Awesome, Ian. Thanks for joining us. **Ian Webster** [43:03] Thanks, guys. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.