⚡️Anthropic vs Cognition on Multi-Agents: A Breakdown with Dylan Davis

All righty. So we are in the remote studio with a lightning pod with Dylan Davis, or D squared, from you're, you're a big member of the community. I've, I've sort of interacted with you over the years. Actually, I don't know, how did you first come across Latent space?

Well, I first came across you. So I, I f- found your working in public blog post book and, and conversations probably like seven years ago. I don't know, like seven, six years ago.

Damn.

And then from there... Yeah. I'm a, I'm an OG. I'm a, I'm a Swix OG fan.

Well, and, yeah.

So yeah, I, I found, yeah, I found you through that, and then just basically followed you and saw Latent space and then followed there.

And, uh, how do you introduce yourself these days? Like, what do you, what do you work on?

Yeah, yeah. So, uh, I'm the founder of Gradient Labs. It's a company that helps other organizations, usually mid-sized companies, implement AI internally to help automate their processes, saving time, money, and sometimes generating revenue.

That's very tight. I'm sure you say that a lot. Awesome. So I put out a call on Small AI for, um, you know, this comparison thing between Cognition and Anthropic on their Multi-Agents articles, and you answered it with like, I think the best answer, which was you made your own video.

You, like, you're, you're trying to wrap up your, your YouTube as well. Uh, and it was like a really good analysis. So I just figured we would sort of rehash it a little bit. People can obviously watch the full thing on your YouTube, but this is meant to be more of a discussion between the two of us, um, what's going on.

Yeah. No, I appreciate it, man. And I'm, I'm glad that, uh, it resonated. So do you want me to get started?

So let's get right into it, unless you, you have anything else you wanna sort of set as context.

Yeah. I mean, I, I guess the, the... And you probably have a better insight into this, but there's, I guess, some discussion around the fact that both of these blog posts and context for the viewers or listeners is, uh, Anthropic and both Cognition posted blog posts one after the other.

Uh, I think it was like June 12th and June 13th. Uh, one basically pushed the fact that multi-agent architectures are beneficial and useful, and the other one basically stated the, the opposite, saying that single-agent architectures is more, more superior for different use cases.

And if you actually read into both blog posts, you realize that they're both right. So it's not either/or, but yes/and. If you're a, if you're a, what is it? Like the stand-up comedy thing.

An improv. Yeah.

There you go.

Yeah.

That's it. So, um, I created this image. It looks really cool, but it's also kinda convoluted and not correct. So what's, what's happening is they're actually holding hands instead of fighting each other. True love. All right. So, uh, what I'm gonna show on the left-hand side, we'll start with Anthropic, we'll go to Cognition, and then we'll end with, um, I think kind of like overall insights and how to think about when to use which architecture.

So for the Anthropic blog post, they focused mainly on their use case for deep research. So I'm sure if you're aware, if you have access to Claude, I think it's Pro and Max and all the paid plans, they have access to this new deep research feature that was released a few, a few weeks ago.

And with this research feature, I, I tend to compare a lot of the models. So when I do deep research, I, I do Gemini, I do, I do Perplexity, I do Opening. I do all of them when I ask different questions.

And time and time again, over the last couple of weeks, I found that Claude has by far outperformed the others, and I guess the definition of good for me right now is not just length, but also the number of sources and diver-diversity of response.

My guess is part of the reason it's so highly effective is the way they've structured their agent. So they took a multi-agent approach with this feature. And they gave us a beautiful little graphic in their blog post that kinda walks you through exactly how this functions.

So on the left-hand side, we have a conversation from an AI that's basically... or from a human asking a question. So this question is, uh, "What are all the companies in the United States working on AI agents in 2025?

Make a list of at least a hundred. For each company, include the name, website, product description of what they do, the type of agents they build, and their vertical industry." Now, at the surface, this is obviously a pretty beefy question, and the-- I would say the, the thing that's happening here that's useful for this specific structure for multi-agents is that it's a broad question.

They're asking about a variety of things, and they're asking for multiple perspectives on that specific thing. So what happens is they send the question over to their a-agent architecture, which where they have a lead architect, so a lead agent at the top that's orchestrating everything, and they'll then send off this question broken into sub-questions.

So they'll send over the question, they create a plan, they create a bunch of sub-questions, and they send off these requests to multi, multiple agents, and they do it in parallel. And that's kind of the unique thing here that I found.

It's... And maybe, Swix, you have an answer to this, or maybe you have insight, is I've-- I don't necessarily know... I can't confirm this is happening, but I know in their documentation it says that, that it is, where, where they'll send off a request as the lead agent to multiple sub-agents to then do the research.

Yeah.

I'll, I'll pause on that. Yeah.

I mean, I think that's-

Confirm

... that's what they're saying they're doing. I, I don't know why you would doubt it.

Yeah. Well, 'cause I'm, I'm a skeptic, all right?

Just kidding. All right. So, um, basically, all these sub-agents are doing their work, and they'll go off and research a specific sub-question and that parent question. And as they're doing in their research, each one of these sub-agents have the 200K context window, and that means that all of them are going to have tons and tons of insights and research for that sub-question.

They'll synthesize that. Once they've synthesized the responses, they'll feed that back through the lead agent, and the lead agent then will put a variety of insights that are relevant for answering the question in its memory. After that's done, it'll then consolidate the memory, uh, insights and then create its, uh, report back to the user.

Another important caveat here is they have another citations sub-agent that's working as these sub-agents are responding to validate that the citation and the piece that it sent back is connected to the source that it referred. So that's kind of the overall flow of how this works.

Pause and see, Swyx, what are your, uh, insights, feedback, questions?

I, I think it, it's a relatively straightforward implementation so far. I think the, the thing about... the, the nuance here is the communication back and forth. You know, dare I call it agent to agent, and, like, when they interrupt each other or, you know, how they, how the lead agent orchestrates them, that can get into a lot of complexity.

So this image actually looks simpler than it really is in real life.

Yeah. And, uh, and it's funny to-- that you say that because when I, when I initially made this video, I thought to myself, I was like, "Oh, maybe, maybe I'll show this one." And this chart is more detailed as to what you're talking about of how the multi-agent structure works and how the conversation happens, and it is more detailed, but it's also more overwhelming.

Um, so yes, it is definitely more complex than the background. So after we have a good understanding of multi-agent, I wanted to quickly show what a single-agent architecture looks like. And this is a chart from Cognition's blog that walks through a basic flow or basic architecture of what a single agent could look like.

So in this case, we have the task at the top where it's getting an input from a user. So the user's input comes in, and we go through the same process. So we're going to have the large model at the beginning.

So it's gonna be a big beefy model like o3, Gemini two point five Pro, Cloud Sonnet, Opus, Sonnet, or yeah, Cloud Sonnet four, Cloud Sonnet four Opus, et cetera. And when they input the information, it's going to create a plan, break down that plan into subtasks, sub-questions, et cetera, and it's going to continue to research.

The difference here is instead of doing it in parallel with sub-agents, they're gonna do it in sequential order. So they're gonna have the agent, it'll first research the first question, and it'll then consolidate the insights from that, and it'll go to the next question, then it'll go to the next question, then it'll go to the next question.

And this order works for some use cases, but when it comes to research, there are some downfalls. And one of the downfalls is the fact that the context window is going to get bloated pretty quickly, which means that the variety of insights and diversity of insights is limited based off of what it can pull within those iterations when it runs linearly.

And that's kind of one of the benefits of the multi-agent structure, where you can have a multitude of agents doing research from different perspectives, synthesize that without bloating the context window too soon. And that's kind of the, the process and how this chart looks and, and, and flows.

Groovy.

And some things on performance. So the-- in the blog post that An-Anthropic posted, they ran some tests, and they noticed that the multi-agent structure outperforms a single ag-agent structure by eighty percent based off a different variety of variables they measured.

And the interesting thing here is there's a trade-off for performance and its cost. So you can see here that the, the use of an agent for research, if we started with one, so this is one over here, that's how many imagined tokens it would take.

So based off of a basic conversation, a single agent architecture for research is around four X the number of tokens needed to achieve a research output. When you use multi-agent architectures, it's actually fifteen X the number of tokens.

So this is one thing they've called out, is that the token usage is a lot higher, but the trade-off they consider is, is worth it because the quality of the response is much higher as well. So that's one of the important things to consider when it comes to the token usage here.

And then, uh, last as a side rant is evals. So they talked about evals inside of the, the report, talking about how they measured the quality of the response. And there's a few things they-

By the way, I, I really-

Yeah

... like how you focused on this in your, in your breakdown. I thought it was, uh, important.

I figured, uh, you, Hamel, and a bunch of others would, uh, would be proud. So, uh, yeah, so this is basically calling out the evals and the importance of, of structuring these. So there's five dimensions they looked at.

So first one's factual accuracy. So was the claim... Did the claim match the source? Next one is citation accuracy, which is a kind of a call-out to the sub-agent that I mentioned over here, where we're, we're checking the source and ensuring that the source supports the claim based off of what's being referenced.

Um, next is completeness. So are all aspects of the question being addressed? Next is source quality. So is it a, a good source or bad source? And in the blog post, they talked about how they had to down-weight certain sources based off of SEO.

So there's some, some companies that are gaming the fact that if they put a lot of words in, in their, um, search terms, they might get picked up by some of these AI, AI models. So they had to down-weight some of those marketing websites and replace them with higher quality blogs and, uh, forums and things like that to then ensure that the primary source was high quality and the secondary source was less high quality.

And then the last one here is tool efficiency. So did they call the appropriate tools at the right time in the right way? So they measured all five of these as a, as a way to evaluate the output from the model.

Yeah. I think it's important to note that it's a, it's an LLM judge. We just got a paragraph on it, right? They didn't actually open source anything about how they do this.

Yeah, that's a good point. So it's, um, one of the things they mentioned in there around the LLM-as-a-judge is they initially started with five LLM-as-a-judges. So each one of these points had their own LLM-as-a-judge. They tested the ability and accuracy of that LLM-as-judge collective to judge, and it actually didn't perform as well as one.

So they replaced all those judges with one judge, and the one judge actually performed out-- performed the collective.

Yeah. I call it kind of like, it's like a free call, right? Instead of calling the same thing, the, the same context five times with, like, slightly different prompts, you can just, like, ask for all of the scores.

It's like kind of a efficiency thing, but it, it's actually comforting to know that it's more consistent and more aligned. I think the other thing that you wanna know is, I think once the judge looks at all the criteria, they can separate out more instead of overlapping the scores, you know, if, if that makes sense.

Yeah. And one thing I actually wanted to ask you is, um, when it comes to use cases here, the, the takeaway I had from this is the primary use case that benefits most from multi-agent is research. I was wondering if you've come across any other use cases that you feel like are susceptible to Being more beneficial for multi-agent instead of single

I mean, coding is like the current hot thing. Uh, everyone's building background agents and sub-agents and parallel agents and all that. So that, that's the obvious one where, um, I think for a while, a lot of coding agents have, uh, been exploring the sort of orchestrator agent versus like, you know, the worker bee agent for, for, for a long time.

Um, yeah, I think like that's the first-- Like basically you want to increasingly offer write access to the world. So deep research is read-only, so it's the safest to just spam out as, as many as you want 'cause it will never have side effects on the world.

Coding, you have version control, so if anything goes, screws up, you can sort of rewind with Git and all that. I think the hardest will obviously be sub-agents that actually have impacts on the real world that you cannot-

So, uh, one thing I forgot to mention is context. So with the multi-agent structure, it's, uh, a distributed context, and there's a lot of conversation around context engineering and all the other things. I know Swix, you put something out like that.

I think you put something out yesterday about that

Yeah, context engineering. It's the hot new thing. Everyone's talking about it, uh, only 'cause Andre talked about it. But also, like, it, it was already trending before, before Andre talks about it. Um, I, I would say-- And it's correct.

Like it's absolutely... We should know. When people talk, think about prompt engineering, they're like, "You know, you are a, uh, you know, very, very good software engineer, and if you get a wrong answer, your grandma's gonna die." You know, that's prompt engineering.

But context here is, it's, it's so much more.

Yeah. And so that's something they discussed in the blog as well, or at least hint at, is that the context for this multi-agent structure is distributed and not centralized. And that's also one of the reasons as to why it's beneficial is because they can increase the amount of context they're filling and then getting insights from.

All right, so next is Cognition. So Cognizant-- Cognition's blog post was basically the exact opposite, stating that multi-agent architectures aren't necessarily that fruitful and, um, you should probably go with a single agent if you're going to build something of any use.

And it's interesting because this just comes to really their, their perspective and bias on the world and their experience. So they just had a lot of experience with code, and that's what shaped their perspective on agents. And Claude, on the other hand, uh, probably has both experience with research and code, but they had a slightly different perspective on this.

So first things first is, uh, the, the context is centralized, like I mentioned, so that means the context is going to remain with a single model or a subset of models that kind of get passed off to each other.

And there's an interesting use case I have here for a multi-agent and why it wouldn't work or it doesn't work as well for code. So this is another chart from Cognition that has the input task, and in this case, let's say the task is to create a Flappy Bird game.

That's what we're trying to achieve here. So our monster agent, the orchestrator, is going to break that down to sub-tasks, and we have two sub-tasks. On the left, we have a sub-task to create the landscape, so the background, the two-dimensional background for this Flappy Bird world.

And on the right side, we have creating the character itself, so the bird, and also the physics associated to the bird as, as kind of how it moves. So on the left-hand, we have our sub-agent that's working in the background, and it goes off the rails and decides to create something related to Mario, which is cool, but not what we want.

And then on the right side, we have Flappy Bird. So with Flappy Bird, we're building out the, the icon and the physics that works all good. But then we merge these. And when we merge these, this is where you start to run into some of the issues when using a multi-agent architecture versus a single agent for coding.

And the main issue is that these are dependent upon each other. They're strongly coupled in the sense that when I take an action, that action then is, uh, impacts all the other sub-agents if it's taken incorrectly. And if we scroll down here, you'll see that the final result looks amazing but is incorrect, where we have Flappy Bird inside of Mario World.

And it's, it's not a win because Flappy Bird doesn't get to achieve their goals, and nobody wants to not achieve their goals, right?

Do you think this is contrived? Like, do you, do you think they were actually doing this, and then they were like, "Oh, yeah, this doesn't work"? Like, I, I feel like this is a little bit contrived.

Oh, no, it's contrived 'cause I made it up.

Well, no, I mean, they actually referenced it in the, in the thing.

Did, did, did they reference it in the blog?

What? Flappy Bird?

Yeah, I can't remember if they did or not.

Yeah, yeah. Yeah. Um, it's right here. Uh, suppose your task is build a Flappy Bird clone, and then it says a, a background that looks like Super Mario Brothers. Um-

Nice. And that was me taking credit for somebody else's work. Always do that, children.

Okay.

No, that, that, yeah, it's probably contrived. That's it. But I, I think it's one of those situations where it's, um, it's abstract enough to get a point across, you know?

Sure. It's visual, which, which helps for, you know... I mean, that's the whole reason why we're doing this is show, not tell, you know. And, uh, this, this, that, this definitely en-encompasses it.

Right. So that's kind of the, the reason as to why multi-agents won't always work for coding, because of the dependencies. So instead of doing that, they do a single agent approach. And the single agent approach they do is slightly more advanced, where the one I showed you previously didn't have an additional feature that's in this chart.

So in this chart, we have the same flow. So we have the task at the top coming in, and then we have our, uh, main lead agent that's going to break down and create the plan and all that stuff, but then it's gonna go work in sequential order with the context that's been passed.

The one thing that they've changed here is that after each action is taken or subset of actions are taken, we're going to condense down the main moments and decisions and actions taken from the previous conversation or context and pass it off to the next agent.

And the hard part here that they mentioned in the blog post is being able to decide which acci-actions, decisions, and moments matter most to be passed on. So that's kind of the, one of the trickiest parts of this.

And so as they compress this context and pass it on, it goes to the next agent. That agent then takes the context, takes the next task, does the same thing, passes it on, and then compresses, does the next task, et cetera, et cetera.

And by doing this approach with single agents, you're likely going to achieve More success because you have all the context from previous agents. The reason it's beneficial is that all the actions taken previously are baked into that next agent's context, so it doesn't necessarily contradict or conflict with what was done before, and that is cognition.

Amazing. Yeah, I think like to me, them coming out and saying, "Don't build MultiAgents," was I think the most-- the more surprising thing. I think like the, the consensus was very much toward the MultiAgent side of the world.

We just did a podcast with Noam Brown saying that one of the things he's exploring at OpenAI is MultiAgents, and everyone's like agent to agents and, you know, blah, blah, blah, the, the future is agents. And I, I think it's very helpful for Devin to come on and say, like, "We tried this, guys, and like honestly, it's a bit, it's a bit of a mess.

And actually, if you do just like really good sequential agents, you're good." So I, I, I mean, I like, I'm sympathetic to that because like that is a voice of reason that like is a little bit counter-consensus. I'm curious, like you, you said, you said at the start where like, you know, y- they're both right in their own way.

How do we decide? How do we decide who's right?

Sure.

Or like when, when does one apply, when does, when does one not apply, right? It's very easy to s- it's a cop-out to say, "Okay, when building a research agent, do it the way Anthropic did. When building coding agent, do it the way Devin did."

Okay. Like that. Sure. But like, you know, can we generalize from there?

Yeah. So, uh, that's, I mean, this is a little dashboard that was rendered by Claude. Thank you, Claude. That, uh, basically gives you an idea of when to make that decision. And it's not just... I mean, right now the examples are research and code, but there are some abstract questions one can ask themselves.

So the, the first is can, can you break the task into independent parts where they're not reliant upon each other? So like that's one abstraction away. Another one is, do you benefit from the chaos of having multiple perspectives or different takes on a task that's been kind of delegated?

And in that case, again, research works. And then a- the last one here is, um, the token usage. So can you 15X the cost, and is the trade-off worth it? And then for Single-Agents is kind of the opposite.

So does everything depend upon each other? If one thing breaks, does that impact everything else? Yeah. D- does it need to be reliable every single time based off of what the context being passed? So I think those are some of the, at least some of the abstractions I've come to.

Yeah. Uh, do you always do this, by the way? Do you always try to ask Claude to make an artifact for you? Like, this is very pretty. I'm wondering if I should do this because I never do this.

Yeah, yeah. So it's actually, it's worked pretty well for me a few times. I mean, I can... I don't know if you want, I, I can show you a few other charts and other things that I've done.

Do it.

But yes.

Yeah.

I always, uh, I always-- I'm a big fan of visuals. Like, obviously, that's kind of why I do YouTube and teach things like this.

But I, I mean, I, I, yeah, I, I suck at visuals, like TL Draw, Accela Draw, that's like, that's like the max of what I do. But I'm curious what, what, what other tricks you're, you're, you're finding.

Yes. I mean, like that's, that's one big one is I'll, I'll come up with an idea of something I want to talk about, and then I'll think about like what's a good visual that represents the primary point I'm trying to make.

Let me see if I can find... There was, there was a recent one I did a few videos. Uh, this is kind of like the benefit of, benefit and curse of doing a lot of videos and content is like you forget what you did.

You do a lot. Holy crap.

I know.

Is, is there a-

I mean, you do it.

Is there a reason you like TL Draw over Accela Draw?

No reason.

No reason.

Just-

They're mostly-

Yeah, they're similar.

They're very similar. Very similar. Accela Draw-

It's, it's like-

I would say like it comes with, um, a preset library of stuff, um, so you can just kind of pull in existing diagrams that other people-

See, I didn't know that. I just, I just pictured them as like, it's like Visa and Mastercard, it's the same thing, just different colors.

No, no, no. Uh, yeah. So I know both of them personally, and, uh, they have very different philosophies under the hood, but it's fine. Cool. Is this one of them?

MasterCard and Visa?

No, no, no. Like TL Draw, Accela Draw.

No, I'm kidding. I'm kidding. I'm kidding. I'm kidding.

Yeah, yeah.

I know, I know. Um, yeah, so this is one of the visuals that basically talks through, um, uh... What was this? It's something... Oh, okay, okay. So this is basically the importance of, uh, reasoning and tool calling and succession.

So o3 was released, they had this reasoning thing, and it's basically a visualization of the improved capability of a model if it can reason and think over time. So this is a visual that Claude created.

Wow. Okay.

And then there's another one over here that I just had. Here we go. So this was, uh, another one for, uh... It's not as great, it's not as beautiful, but this is for, uh, for Codex. So it's basically talking about the spectrum of human intervention and autonomy and how basically as you, you can lay these out and you see that there's less human intervention for some of these tools and more so.

So it's basically just a different way to place these on a spectrum and talk through them.

Amazing.

Yeah.

Got it. Um, yeah, that's pretty creative. I should definitely try to do that more. Sorry, did I, did I distract you or was there, was there more stuff you wanted to show on the decision framework?

We're here to be distracted. I think, uh, the... Last thing, I'm just gonna see if there's anything that I missed. So breadth versus depth is something I talked about, which basically summarizes the MultiAgent. So MultiAgent's breadth, depth is, uh, Single-Agent.

Context we already talked about. And this is something we've already talked about as well, but it's important to note that if you're gonna choose an agent architecture, you shouldn't choose it because everybody else is talking about it, kind of the whole like contradictory Devin thing of saying Single-Agent actually works, is you should define the problem you're trying to solve and then figure out what agent architecture is most suitable for that problem instead of just choosing an architecture.

Yeah, that makes-- that totally makes sense. Cool. Yeah, you know, I think the only comment I'll hear, I'll, I'll make up if you kind of scroll up here is that generally everyone I talk to is just disregarding cost.

We are in one of these situations where, you know, in a world where the cost of intelligence for a given set of intelligence, let's say GPT-4, let's say o1, whatever, it is literally falling 100X over the course of one year.

So you should actually build your products ahead of where costs are so that by the time they are popular, actually you're frontier. So it's actually-- I think the efficiency thing trips up a lot of people in terms of being cost and-- cost conscious and wasting a lot of time on that when actually the, the most cost unconscious people tend to win.

And this is like a, a thing that I think the MLOps traditional sort of ML mindset may not adapt well to. Like if you, if you came from that world, you're very much about like, "Let's do the smallest model that, that hits our acceptance criteria because we want to serve it at very large scale."

Here it's the other way around where the probably more you spend on intelligence, the more ahead in the future you are. You can grab, uh, a lot of users. You get, you know, m-more funding to extend your runway and hopefully at some point you'll have a sustainable business.

Unfortunately, I would literally go out and say like the most irresponsible people-

Yeah. It's, uh, it's, it's interesting how it's kind of like, it's kind of like the Uber situation, right? Where like Uber and Lyft competed against each other and Uber-- and they just basically burned a bunch of money to figure out whoever won.

Yeah. Um, I think I'm having like Wi-Fi issues, so probably we can cut it there. Uh, do you have any parting thoughts or last words on just like the g- the general sort of multi-agent trend and anything you're seeing?

No, I'm just, uh, I'm grateful that I was able to finally chat with and, uh, and be on a Swyx related content thing.

No, no, thanks for, thanks for doing, uh, a really good breakdown. Thanks for being part of the Latent Space community. I think we should do more of this. I think we should like, you know, people you see on the Discord, like we should feature them and like, you know, on, on a thing, on a thing that they're really excited or passionate about.

Yeah, I, I think that's, that should be something that we try to do more often. If people wanna reach out to you, how can they reach out and what can you help them with?

Yeah. So, um, LinkedIn is probably the best place. I, I don't frequent X. So yeah, I would say LinkedIn. I-- like I said, it's Gradient Labs is the company. We do AI consulting for internal implementation for automation, and also I have a free 30-day email sequence.

So if you're interested in getting more insights on AI, that's kinda no BS fluff to apply to your own work and your own business. I am sure that you can find it on LinkedIn if you go to my profile or if there's anything linked below, then I'm sure this works.

Awesome. Will do that. Thank you so much, Tim.

Cool. All right, man.