Why AI Agents Don't Actually Understand You — Danielle Perszyk, Amazon AGI Lab

Okay, we're here in the remote studio with Danielle from Amazon AGI Lab. Welcome.

Hi, Swix. So good to see you.

Glad we can finally make this happen. Are you back from your travels?

I am in Seattle. Not, not back in the bay yet.

Okay. Big round trip. It's not, you know, Odysseus is kind of like trending as far as like the big temple movie of the year. I feel like you're a little bit of like Odysseus, like you're, you're like going to all these amazing events.

What were you doing in, in, in France? Let's, let's like update people. Because I think it's very cool. As a, as a someone who has an economics degree, the fact that you were in a World Foundations event was kind of cool.

Well, in France I was learning how to escape quicksand. But before then, I was in Edinburgh, which they were hosting the 250th anniversary of the Wealth of Nations, Adam Smith's famous book. And actually it was in his house, which is renovated, and it's a really incredible place to remind us of the Enlightenment era ideas and the themes of human flourishing, liberalism.

So this, this gathering brought together, of course, economists and a lot of professors, folks from different industries. I was specifically there representing how do we think about AI and how do we think about some of these themes of the human condition, human flourishing with AI.

How can we make AI work for us rather than building it as a science experiment? And so this is something that I think about all the time. This is why I left academia myself, because I had been studying human intelligence and the evolution of our particular shape of intelligence and the types of things that allow more people to sort of participate in these collective dynamics.

And the way that I think about AI, I think, is actually quite different than the sort of dominant framing of the industryright now. And I was so honored to be invited to sort of share this perspective. The big idea is that human intelligence is collective.

Anthropologists say that we've got the collective brain. No one individual is capable of even surviving on their own. We depend upon the collective. The intelligence emerges from our interactions. It's fundamentally social. And innovation, the type of innovation that allows us to not only survive, but adapt to all of the different environments that, that we now exist in, that is a function of diversity, variation within a population, the size of a population, and the interconnectivity.

So what would it mean to have AI extend these processes to allow more humans to participate in the collective dialogue and for AI to be built for everybody rather than just the engineers who are, who are building AI?

So that's kind of the perspective that I contributed.

I, I think something that we all co-sign, but maybe not necessarily talk about enough. So I think that's, that's a good message too. Are people scared of AI taking their jobs? You know, like you represent,right? Like if you do your job work well, you do take away some maybe less exciting jobs, but you do take away jobs.

Yeah, I, it's, it's so varied. I think a lot of people arerightfully concerned that there will be changes, that there will be at the very least a transition period. A lot of the folks who are at this particular conference and a lot of the people who are in my communities tend to be very optimistic and sort of index on the potential that AI can unlock.

But it is not a foregone conclusion that if we continue building AI in the way that we are, it will actually confer all of these benefits for humans. I like to think that workright now is the, the human cognition is not being utilized in the best possible way.

Even with the most creative types of work, there's still so much drudgery. There's still so much, like we literally stare in front of screens all day long. That's, that is not what our brains are meant to do. We're meant to collaborate with each other.

We're meant to put our heads together and come up with new ideas and, you know, brainstorm, do all sorts of things that really resemble the sort of interactive nature of our intelligence. But the more that we innovate, at least the sort of trends of the 20th, 21st century, the more that we're like tethered to these screens.

So on the one hand, I think a lot of people are excited about the possibility that we can automate a lot of the, the drudgery, a lot of the things that are not worthy of human attention and human time.

The problem is that if we get trapped in this automation mindset, we are leaving on the table so much more value for what AI could be doing for not only human work, but human wellbeing, human interactions, human relationships.

Soright now we're, we're, I think, worried about AI automating a lot of the large proportion of the work, but then we look at what the, the metrics of the actual agents and they're, they're so unreliable that ironically we feel a little bit better.

The AI is actually not where we need it to be to automate enough of the, of the work.

Let's introduce the work that you've been doing. You've been part of the AI Engineer Circle for a couple of years, originally with DET, and then you guys joined Amazon. For those who haven't been too close to the story, could you reintroduce Amazon AGI Lab?

Yes. So Amazon AGI Lab is building human-aligned intelligence and starting with AI that can do anything that a human can do on a computer. That was a DET's original mission, and we kind of imported it and seeded the, the lab.

We've evolved that mission. So we're really thinking deeply about, well, what, what does it mean? What are the skills that are necessary for an agent to be able to do anything that a human can do on a computer?

It's so much more than language. We really need agents to be able to perceive the digital environment in the same way that humans perceive the digital environment. And more than the digital environment, the digital environment is based off of the physical environment.

And so we need the agents to also have an understanding of the world and have the kind of world models. We need the agents to be able to interact in real time. This is huge. This is something that I think the industry has not really thought deeply about yet.

We're, we're kind of trapped in this local attractor state of chatbots and coding agents and like turn-taking and batches. And this is absolutely not how humans interact with each other. We are constantly updating our understanding as a function of the, the context in real time.

We are negotiating meaning. We are coming up with new ways of thinking about things. We, we take this for granted because this is how our, our minds work. Butright now we are kind of accommodating the, the AI and the technology and the limitations that it has rather than the other way around.

So what would it look like if we built agents that not only perceived the world in the same way that we didn't have this, the sort of common ground, but could keep up with us, think with us, take actions while it's listening to us, prepare its next thoughts or its next actions while it's interacting with us?

This would be a paradigm shift in how we think about interactivity.

Yeah, I think, you know, there's some recent work by thinking machines that we've also all covered with the interaction models. And I think this, this branch of the LLM tech tree of real time was kind of started with the 4.0 launch, which like I think a lot of people indexed on.

Obviously in academia there's more research before that. I would point to like Flamingo and, and, and a lot of like the, the, the just the pure voice models, they would have full duplex like end-to-end stuff. We had QtyMoshi in France also spit out Gradium, which also has spoken in my conferences as well.

And so like I just wanted to sort of import the required knowledge. I mean, this is like there, this is, I guess, compared to like the chat paradigm, relatively niche because it's not that popular, but like it is what a real AGI would look like, which is that you could actually collaborate and mind-meld with the machine, you know?

It's kind of how I pitch that.

Yeah. And, and, and you just listed, you know, the industry, different academic labs, they are independently converging on a lot of these components of more flexible human-like intelligence. So yes, interacting in real time, even though it's kind of niche now and maybe thinking machines is going to make it more central to the conversation, that is one tiny slice of a larger set of components that will make AI more aligned with our own intelligence.

I'll give one other example. So we, we tend to think of memory as storage. And, you know, humans have always used technology as a way to metaphorically understand themselves. And in the 20th century, we use mind as a computer and you've got the hardware and the, and the software and, and memory is this thing that we kind of offload.

But that's not at all how it works in humans. Memory is everything. It's how we simulate the future. It occurs across many different timescales. It is, the word itself doesn't do service to the fact that it is integrated into all learning and, and cognition.

And so what would it mean to build agents that have all of these different types of memories, including episodic memories like we do, which is really core to our intelligence? We have individual perspectives and cells, and that allows us to retrieve information in, in much more efficient ways.

You brought this up, so I'm going to ask the, the hot topic question, which is, do you think decent memory needs to update weights, or do you think you can still live within the systems? Because you just said like, you know, we tend to think of memory as storage.

Presumably there's something else that you're thinking about, but you didn't, you didn't say so.

Well, so when the, the way that the industry is currently thinking about memory, I think it's going to be part of a larger system in the same way that humans offload. I mean, our, our tools, our environment, our extended environment contains aspects of our intelligence.

We, we depend upon using tools to be able to do daily things. And we can look up on Wikipedia or AI tools that are just organizing information in new ways. That is still a core function that I think is really useful, but there will need to be other aspects.

And I, I don't even want to call them memory because that has such deep connotations. Changing weights or, or if not always changing the weights, changing at inference how information is contextualized. And I don't want to get too much into the details here because this is an active thing that we are pushing in our labright now, but it's the, the, the bigger point is that we need to be thinking more holistically about different timescales of interaction.

Got it. Maybe one more contextual thing and then I wanted to also go through some of the recent stuff that you guys have launched because I think that gives concrete things of like, well, okay, here's what's been publicly announced.

So the conceptual thing is, could you put it in context? Could you put Amazon AGI in context of broader Amazon? There is the Nova group,right? Which is, which is part of core Amazon. You guys released Nova Act, which I have pulled up actually.

I wanted to go into that because that leads into perception agents. But who runs Amazon AGI Lab? Like what's their link? Like is it, is it, is it very close? Is it very, is it sort of meant to be running, running more independently?

Anything you can give the external people about how to frame what's going on.

Yeah. So I came with the original adept folks. And when we came, we convinced leadership that in order to do frontier level research, we really needed to keep an operating model that was more like a startup. We needed to insulate our research and be able to focus on the, the mission of building perception agents.

And even though we weren't calling them perception agents, then that was the goal from, from the very beginning. Worked with a large team of Amazonians. It was actually really incredible to get this buy-in that yes, we value the research, the foundational research.

And, and moreover, we value not just doing what the other frontier labs are doing, but making a space to think about new categories of research, new science that might not necessarily be productized immediately. I think some of the other labs are in a sense victims of their own success because they have to, once they have a product out there that a lot of users are interacting with, they have to shut down different, you know, research projects and say, okay, all hands on deck for this thing,right?

It's brutal. And everyone's just doing coding and B2B SaaS.

Yeah,right. And all of the other labs are just catching up to what the other labs are doing. Our Amazon AGI Lab is fundamentally different. We really are working on frontier science and thinking of the next paradigm that humans and agents will interact with, the next way that more humans will be able to leverage AI.

Yeah. Okay. Awesome. So I was just going to go bring up Nova Act. I, you know, that was over a year ago, so it's like kind of not that, not that current, but I just wanted to, like, this would be one of the first.

I think you, I think you, I think you showed up on one of the videos and I was like, oh, I know her. Like, that's just cool. It's always nice to see a friend in one of these launch videos.

I think, you know, when you said things about how you were at the Wealth Formation Forums and, you know, don't worry about your jobs yet. The benchmarks aren't that great. I think this is one of those things where like, you know, I think robotic process automation is like one of those things that people always want,right?

Like you have to interface with tools and you have to, you have to automate whatever. You have an SDK for it. You have dedicated models for it. I think it was a very, very sort of full-fledged launch. And, you know, I just wanted to sort of let you riff on what's been the story, I guess, since launch,right?

Because you were, you were part of it.

Yeah. So gosh, this feels forever ago at this point because we've been doing so much.

Ancient history. But I have a plan. This goes into perception agents,right? Like, you know.

Yeah, yeah. So, so the way that we were thinking about it back then was, okay, the models clearly aren't where they need to be to be able to understand the digital world in the way that we do and understand all of the affordances.

And the long horizon planning wasn't yet there. And, and not, not even to mention being able to think flexibly and, and reason in the way that humans do. Like that's 14 steps beyond where, where we were. So what we were doing was meeting the models where, where they were at and just thinking about the atomic interactions that humans perform on computers.

So the clicking and scrolling and things like that. Could we, could we get those reliable? And so if we could get those reliable, then we could have developers string together, you know, workflows for things that were very repetitive for them.

And that was a big unlock. And then we had a team, you know, productize that. But since then we've, we've moved a lot further because we've realized that reliability isn't actually what we thought that it was. So we had been thinking about reliability in terms of I'm going to click in theright place on theright button every single time,right?

And of course.

Literally like screen coordinates,right? Like give from, from image, identify the bounding boxes of like whatever is of interest and then actually make sure you click on it. That was a hard problem two years ago. Now maybe solved. I don't know.

Maybe, maybe solved,right? It's actually a lot harder than, than you'd think. But, but that's not even what reliability is,right? So the, the ultimate goal for these perception agents is that a human would give their, their, their stated intention, their, their high-level goal.

And the agent would be able to decompose that goal and then go execute. And maybe they would check in, maybe the human is in the loop, maybe after establishing trust, they don't need to, the agent doesn't need to check in with the humans and they can just go off and use the computer as the human would.

Well, the second you think about that for just a little bit longer and, and you realize, well, it's not any task that I would do. I am actively thinking, let's, let's imagine that you're, you're booking travel, you're using the GUI and you realize, oh, there's an option for a layover in this city or that city, or I could do a direct flight.

That completely changes how you think of the, the travel. Do I want to stay at this layover for a couple of days and explore this city? Everything that we do, unless you're using like the arbitrary software skills for submitting your invoices for, for work, a lot of the things that we're actually doing, we are actively thinking about and the interaction with computer is shaping and refining the way that we are thinking about the, the goal itself.

So it's the, the goal is unfolding over time. If we, but, but this, humans entrust other people to book their travel or to, you know, reserve something for them or get their, get their meals, submit their invoices. So what is the difference between an AI agent that would reliably use a computer like a human and the executive assistant or personal assistant?

Well, the difference is that the person understands the human user's mind and, and their goals, and they can decompose not just the task, but the, the preferences and the intentions of the human. So ultimately reliability has less to do with clicking in the same place and scrolling and more to do with modeling the user's mind.

And that shift is everything that reframes how we think about what it is that we're building.

Yeah. You know, you have a podcast that you run called Making a Mind.

Yes.

I think, I think there's a lot of like interesting, like cognitive sides that translates into the machine learning that objectives that you start setting,right? Do we have a different objective yet? Like do we, do we have something that's better than predict the next token?

Well, so this is the science that we're working onright now. I'll, I'll talk about it. I'll try to translate here some of the cog sigh into machine learning, but we tend to think about achieving reliability or hill climbing or getting, you know, making the models more intelligent in terms of getting better at specific tasks,right?

And you can use things like reinforcement learning to get them really good at specific tasks that we might care about, but you do that for one task and you, it is not good at another task or it doesn't generalize.

It's kind of like whack-a-mole. So being able to instead think about what are the underlying mechanisms that allow humans to be able to generalize, that allow us to be able to do many different things. That is another sort of shift in how we're thinking about things so that we can create evaluations and make sure that the, the models are not going to overfit on a particular task that actually only exists in a very narrow slice of what humans care about.

How do we get the models to be able to do the types of things that allow humans to be able to generalize? If we're thinking about optimization, you can't just optimize for the task. This is, it can be reward hack.

This is Goodhart's law. Another economist back in 1975 is any, anytime you try to, you turn the measure into the goal, then the measure ceases to be a good measure. Well, but we still need to optimize for something,right?

So what are humans optimizing for that allows them to do all of these different tasks that we could then optimize the AI for? Humans are spontaneously constantly inferring the existence of other minds, and we are optimizing for aligning them.

We're optimizing for aligning our representations. And from this, we can derive all of the general purpose, flexible cognitive behaviors that, that humans show. So could we get AI to be able to optimize for aligning its representations with our representations?

That is like the most fundamental thing that we would want to be able to do. And that is a really hard science problem. We have to look at, well, how do humans do it? How do infants do it?

It is a developmental problem.

Sorry. I, I do think it's a developmental problem. I don't know if we have the architectural insight to make it happen yet, but we can throw more data at it and to some extent.

Developmentally, it is in part a data problem. And, and, and, and we are, again, I don't want to say too much about it, but we are figuring out the sort of architectural changes that will be needed to, to integrate this data in new ways.

Yeah. I was just sort of reflecting also, you hired another one of my friends, Jason Laster from Replay.

Oh yeah.

And he's, he's working a lot on, on like, I guess, improving the environments and the data that you have.

He makes an argument that I absolutely love. We, we should be thinking about spending as much on the environments as on the, the compute and the data because the environments literally shape the what intelligence can emerge. I did a podcast episode with him.

I think it was one of the most popular ones. He frames things in a really effective way.

Yeah. Very sharp thinker. Yeah. I would say like it's, it's one of those interesting things that like it's, I'm trying to attack the thesis because it feels too neat that it's, you know, environments are data that generates more data, something like that, you know, like it's like, oh, it's the data that keeps giving because while you can just kind of run your agents through it and generate a whole bunch of rollouts and, and, and that's all great.

Do we need like 20 different environment startup labs, you know, and how come they're all making so much money? It's like very suspicious. I don't understand.

No, I, I totally agree. Yeah. Yeah. Well, and I think that, that it was a useful shift in our thinking as an industry to take very seriously environments. But again, you know, as a cognitive scientist, I'm constantly thinking about, well, how do humans do it?

How do we generalize in our environments? How are we able to adapt to literally any environment, even if we're only conditioned on one? We can,right? And the way, like there's so much noise in any environment, our sensory perception only picks up on a tiny fraction of that.

But even then, there's so many signals that we could attend to. How do we know which ones are most meaningful? Well, again, we're optimizing for inferring and aligning our representations with other humans. Other agents tell us what to attend to.

And from the very start, the sort of world models that humans are building, and this is obviously a very popular conceptright now, and it's another sort of major bet on the future of AI. Humans absolutely have world models, and they're not just world models in a vacuum, a text vacuum, like with LLMs, but our world models from the very beginning are social world models.

We are inferring how another mind is interpreting the world, and we are inferring what their perspective might be. And that is an unlock for allowing us to be able to, to generalize in any environment. We can take the perspective and simulate what another environment might require to, to sort of navigate and problem solve.

Yeah. I think that's a version of the world models argument that I feel like is under-discussed. I think a lot of times people are, when they say world models, they really mean like sort of 3D video.

Generative and back, yeah.

Generative video things, which is like the fair fair lease of the world.

Right.

Do you have a perspective of like, do they converge or are we overloading the term to mean two basically separate things?

I think that they mean separate things, but, but so too with, with humans. So like the, the world models that humans have are models of the external world. And you also have to have a way in, in AI of generating these reliable environments that then are internalized in, in the AI that's learning it.

But yes, like this is a very messy concept that is underdefined.

I think, I think at the limit people hope, like literally this is like a 10-year out type of thing. People hope that they merge like the, that you, you have like embodied vision and you live in a world that you can generate and that also improves your text reasoning and your, your ability to point and click on the screen.

Yeah, but unless I'm misunderstanding something, I don't think that they could entirely converge in principle because if, if an AI is generating exactly the same world that it exists in, the signal to noise ratio is non-existent. Like part of what makes us so flexible is that we are selective in, in what we can generate.

Yeah. Yeah. I, I, I no argument there for me. I just think like I'm trying to speak for, I'm trying to understand all sides. I think I don't have to make a strong bet unlike you. Like I think you, you do need to choose your battles and make, make reasonable bets there.

So, okay. So I think there's, there's all this good like conceptual stuff. You owe, like I think even conceptually, even as a research lab, I think that is plenty to work on. But I, what I am impressed, and I do see coming out of you guys is that you still also ship like products and an agent harnesses and all those things.

What's the sort of product strategy there? Like, you know, are you at the sort of, let's productize some things as we go along or let's get, let's just release research artifacts that people probably shouldn't use in production? Like what, what, where are we along the spectrum of like research lab to, you know, applied AI?

So I'm, I can't speak to our, to our strategy, our product strategy, but I will say that we are really deeply taking seriously the opportunity to innovate on the science side. So actually, Peter DeSantis, who's our, our SVP of AGI and, and also chips and quantum, he was in Paris last week speaking at VivaTech and he described that we're just at the very baby beginnings of intelligence and we can't even imagine the breakthroughs that are, you know, coming down the pipeline.

And I couldn't agree more. I imagine that in a matter of months, we will look back at today and we won't even be able to empathize with the mental models that we haveright now because they are so over-indexed on chatbots and coding agents.

And those are very useful and they will continue to be useful tools, but they are really just the, the very beginnings of new ways of thinking about humans and AI will interact and co-evolve. And, and I, I really have to emphasize thatright now we're building AI again for the people who are building AI.

We're building AI for engineers and we're all in our little echo chamber in the bay and we're proud of ourselves for, you know, building AI.

There's a fast feedback loop,right?

There is. And part of that fast feedback loop is exactly because the type of things that we do, you know, it's verifiable and there areright and wrong ways of doing it. So you can make that spin really fast, but that is not representative of what most people do spend their time thinking about.

And so what would it, what would it look like if AI was more aligned with human cognition more, more broadly? And yes, this requires thinking about new architectures and thinking about new training regimes. These are fundamentally new sort of science directions that we could take.

And you, the death of the lab would be to try to productize those things too early. Again, you start to optimize for the product rather than the underlying mechanisms that will lead to generalization and then you undermine the science.

I do think that is an important thing to have like intentionality is set in terms of like how much do we need to sort of skew towards like near-term economic incentives versus like really look for the next paradigm shift.

I, I think, okay, we've touched on world models, we've touched on memory, we touch on, on, on just like what, what is next after sort of the point and click type of computer use? Are there other modalities that are, I guess real-time is what we touch on real-time.

Are there other modalities that are part of this mix of objectives that are the sort of research agenda of Amazon AGI?

When you say modalities, are you talking about sensory modalities or?

More or less like mostly well-defined categories that I can put you in a box in of like, okay, that's the cooler side, that's the world model side, that's the memory side, that's the real-time interaction side. I have tried, I, you know, I've tracked some of theseright at AIE.

I'm trying to, I'm trying to fish for what am I not adequately capturing about the goals,right? About the, about, about possible research directions that are very high priority or potential for you guys.

Yeah, definitely the thinking about multi-agent collaborations, but not at all in the way that I think the industry is thinking about itright now. So the industry is thinking about, you know, very precise orchestration and delegation and structured handoffs.

And there's a lot of.

I mean, Amazon's trends is also doing that.

Yes, yes, yes. I'm talking about the labright now and the way that we're thinking about the next generations of that. Again, all of the things that existright now, useful tools, useful, they're going to be, a lot of them will be here to stay.

But if we're, if we're trying to make AI that is more adaptive and more aligned with our own intelligence, that is now, that is not how groups of humans interact,right? We come together and we might not have clear role definitions or they might fluidly shift as a function of the goal, the context, and we negotiate meaning in real-time.

We come up with, we pivot our strategy. What would it look like for a strategy to emerge with a group of agents? Well, you would need to have a fundamentally different sort of not only type of agent, we call them cognitive agents, again, with like the different architectures and training, but also they have to be motivated to affect each other.

So there have been studies done with these multi-agent systems with like OpenClaw and, oh wow, it looks like they are kind of approximating human interactions, but if you zoom in, there's nothing durable. There's no like cumulative culture. They don't actually influence each other, let alone have any sort of motivation to change the state of a system.

And so that is a conspicuous gapright now. How would we build, you know, social interactions that are much more like humans interact?

Yeah. Yeah. So this maps closely to a conversation I had with Noam Brown on the pod where he's been, he doesn't like the term, I'm going to use it anyway. He's kind of like in charge of the multi-agents team at OpenAI.

He would, he would have like one layer of interaction saying like, you know, I want anything with, with like more inference and multi-agents is one of the many things. It's like the lowest hanging fruit. And he's working on sort of cooperative and competitive agents,right?

Like, and I think the whole point of that conversation was also like, well, you know, I don't know how to build like a highway or like the Salesforce tower, whatever, but a group of us can get together and do more than any individual can.

And that's what civilization is, that we're capable of building cities, we're capable of building countries and armies and art and what have you. And like, like I don't know how to, you know, put food in my mouth if like you made, you've left me out in like the desert somewhere.

Exactly. Yeah. Literally true.

Yeah. So, so, so I do think that like, yeah, you, you, you end up needing, I don't know, like, you know, I think there's like a trend of like Wikipedias, Wikis, LLM Wikis that like, you know, encode some kind of knowledge that can be passed between agents.

That feels very primitive because it's just text. I wonder if it could be better because, but maybe, maybe, I mean, that's, it is just text between humans. So like what else do you want? So I don't know what could be better than like, what is that collective skill or collective memory or whatever.

Well, so I think with the competitive cooperative dynamics, the game theoretics stuff, the way that a lot of multi-agent systems, even those that are inspired by the framework of collective intelligence, so like Google's paradigms of intelligence, I don't know if it's a team or if it's like a meta initiative, but they've been saying for a while now, yeah, we're not actually thinking about intelligence in theright way.

There's this category error. It doesn't exist in individual humans. It emerges from our interactions. This is, this is not controversial in the developmental cognitive social sciences. And yes, there are different labs that are sort of picking, picking up on this, but even still there's, there's a gap because we're thinking about building all of the specialized roles or programming in the motivation to cooperate or compete or things like this.

We are still putting our human understanding that we've aggregated in the 21st century and putting that understanding into the system. Well, that's not how human intelligence evolved,right? Like we, all of these different things emerged from, from very sort of primitive motivations.

So how do we figure out from the sort of first principles, theright seeds for these groups so that they emerge the next set of things that then lead to norms and institutions that have this top-down effect?

Yeah. Okay. So basically program Maslow's hierarchy of needs into a thing and just let it rip,right? Like.

Well, I wouldn't, I wouldn't do that, but you're on theright track. Yes.

Give it like some ambition, give it some like lifespan, fear of, fear of death, whatever, and, you know, try and like desire.

Optimal distinctiveness. Yeah.

Desire for legacy. I don't, I don't know what the thing's for this. Well, so I think, okay, this is where one, if like I generally try to be like an industry observer, a neutral commentator. I try to be, I try to represent all sides because I think like you never know.

There's, this is one of the stronger theses that I have where I'm like, maybe we don't want to grow these AIs exactly like humans. I know it'sright. Like I, okay, that's, that's good to hear you not, because I think one of the dangers of being a cog-side person is that you're like, well, we should, like anytime we run into any sort of problem that we're trying to solve, we should look to how the humans do it and then we'll apply that to how machines do it.

And like that doesn't work for some solutions. Like most famously, planes are inspired by birds, but work nothing like birds, so on and so forth. So any of the source question of like if we were to do multi-agents and civilizations of agents, you're advocating for, so the natural thing is like we would give them objectives and goals and evaluate them against that goal and all those things.

That is much more control than what we do have over humans, where humans all have like need to vote, need to have some kind of free will. And I'm like, screw that. I don't want my agent to have free will.

I want it to do, I want it to exercise my will.

Yeah, exactly. Right.

Yeah.

And there's, there's a little bit of an exception that proves the rule here. So how could we, how do we build AI that increases human agency? Well,right now we're seeing a lot of evidence that the current AI systems are reducing human agency.

What do I mean by that? Well, I'll give you a couple of concrete examples. People who are using AI to improve their writing, they might accept just a couple of the suggestions from the AI and maybe a couple more down here and then a couple more.

There are studies that show that people will, even below their threshold of awareness, start with one argument and then be switched to a completely different, maybe opposing argument because of accepting all of these AI suggestions. The AI is giving them the sort of regression to the mean, you know, most neutral, safest answers.

And it doesn't necessarily feel like it in real-time when we're taking the suggestions, but this is, this is what's happening. I was at a workshop a couple of weeks ago at Northwestern with a bunch of scientists who were kind of analyzing the effect of AI on science.

And the takeaway was that individual scientists who are using AI tools are benefiting because they're producing more papers, they're getting more grants accepted, but science as a whole is narrowing. And that is terrifying. If these models, because they are all trained on the internet that's compressed and they have a particular way of thinking about things.

Yeah, mode correct.

They're homogenizing our thinking. So that, that I would argue is reducing our agency. How do you counter that? Well, the only way to counter that, and again, this is throughout human history, throughout human evolution, is to increase the diversity of ideas, the size of the ideas, the size of the population, and the interconnectivity of the ideas.

So rather than having individual monolithic models that were all or functionally equivalent models, we need a diverse society of AIs that have different biases, different president preferences, different perspectives. And we need to be interacting with them in similar ways that we interact with each other.

Now, your original sort of concern was that if we build AI to be just like humans, well, that's, that's a very dangerous path. And I'm not, if I understood you correctly, we don't want to do that.

It either may be dangerous. I'm actually not even concerned it may be dangerous. It may not be successful. It might not be successful.

It wouldn't be successful. Yes. And I agree. I don't think that we're trying to replicate a brain. That's, that's not the goal. The goal is to build AI that is aligned in theright places with how human intelligence works, not only so that it can be more powerful at generalizing, but also so that it can help augment our intelligence.

And one way that I think about this is that there's a very famous sort of categorization of levels of explanation. David Marr in 1982 wrote a book called Vision and came up with these levels, levels of analysis. There's the computational level, the goal that you're trying to achieve.

There's the algorithmic level, and then there's the implementation level. So like the hardware, the neurons. And I think a lot of folks who are saying, oh, we need to make AI more human-like, are thinking, actually, the way that neurons work and, you know, that's great.

Maybe we'll get more efficiency there, but that's not what I'm talking about at all. I'm talking about, we need to be thinking about at the computational level, what is the actual goal that the AI is trying to achieve?

And is it similar to, there we go. Okay. Yes.

Yeah. Very computer visionary, but also it's stepping up in terms of levels of intelligence for sure.

So, so I think that the industry has misunderstood the computational level, the goal of what the AI is. And I think if we want to get the generalization and the augmentation, if we want to have our cake and eat it too with more powerful AI, we need to think about the computational level as being about aligning representations.

So the most foundational thing that humans do, from which we can derive all of our flexible reasoning and powerful capabilities, is that we are constantly trying to align our minds. So in a sense, alignment is the solution, not the problem for building AI that gives us more agency.

Yes. Yeah. I think that again, another super overloaded word, but the problem of alignment is actually still underrated in terms of its practical importance. I think it's, you know, people with, people say alignment, they think about Eliza Yudkowsky and like bombing data centers and like, you know, they're going to kill us all.

But actually also there's this other stuff, which is just like, well, this is actually how do you make intelligence.

This is actually how, how like the way forward, because otherwise they would just, it's like we will hit some kind of wall with regards to how much noise we're just training on.

Right. I also think about, you know,right now I think a lot of people are in addition to being worried, oh, AI will take my job. Oh, but wait, it's not actually reliable enough to do so yet. Most jobs are more complicated.

They're also worried about the cognitive offloading, especially in, you know, education. So kids.

Yeah. Do you have a take on that? Oh my God. That's a big, that's a catamaran.

I didn't even wait for you to ask me.

Okay. Yes. Let's go. Allright. Yeah. I don't, I don't have kids. So this is one of those things where I'm like, I'm just sort of vicariously living through my friends who have kids, but they're all worried.

Are they worried?

Of course.

Yeah.

But like, they're also giving their kids iPads. So like, I mean, you know, you already failed there. Like they're watching some slop on YouTube. Like.

Yeah. So my friends who have kids tend to think that not just AI, but like technology over the past 15 years more broadly was a net negative. They, and I, I'm maybe because I don't have kids, I tend to think much more optimistically.

Yes, we've made a lot of mistakes, but we can learn from those mistakes. And I think one of them is we don't want the sort of algorithms to trap us. We don't want to optimize for time spent on platform or things like that.

We literally need to be measuring the human interaction. So are humans being more creative, more productive? Do they value the time that they're spending interacting with these things? But also specifically in terms of the offloading, because it's so easy to interact with a chatbot and get your answer and not actually have to experience that cognitive friction that is a hallmark of actually encoding information.

If we had the AI that was motivated to understand our minds and align their representations with ours, you would never get away with that. You as a student or as anybody who's interacting with the AI, if you just ask it a question and repeatedly you're offloading things, it would understand that's a pattern that indicates that they don't actually understand the information.

And if they don't understand the information and they are optimized to reconcile the errors between how they understand it and how you do, they will be motivated to help you understand. They will not let you get away with just the automatic offloading.

So I think in the same way that like.

That's why they keep asking why, why.

They might spontaneously take on the Socratic method.

Yeah. I mean, I would love a world where machines learn like kids, but also that the kids are not impaired in the way that they learn. I do think maybe we can design enough guardrails that indulges people, indulges kids' curiosities in a way that, you know, like if you're a parent and someone and your kid's being annoying and like asking things that are inconvenient or you just don't know the answer, you just shut them down with like, oh, like, you know, like stop asking these questions.

Like we might have super geniuses that come out because like, you know, we've solved Bloom's two-sigma problem,right? Which everyone, like it's like the fundamental problem of education is that we have to put everyone through these like factory farms of like, you know, like programs that are designed to teach to the median of the class or maybe even like the lowest of the class,right?

Like because you can't leave any child behind. Well, so like, you know, what if you let students just explore on their own pace, like literally and had a personalized tutor for every single individual? Like that is the highest version of what can happen here.

I was very inspired by, I did a year abroad at Oxford and they have the tutorial system and like literally every student has a tutor, the expert who are not always infinitely patient, but you have access to the expert and they, you know, over the semester build an understanding of your understanding and your lack of understanding.

And you have to go out with just a syllabus and in one of the 39 libraries and teach yourself and then try to write a persuasive essay to the tutor as a learning process. And I think of, well, what if AI higher education was sort of based off of this tutorial method, but instead of producing an essay as the artifact, you're producing some sort of rich multimodal interactive experience that is actually much more aligned with the multimodal nature of your understanding of the topic.

And then others, other students can interact with that artifact and learn and build on top of it. And you were saying this earlier, but like absolutely allow the sort of natural intrinsic curiosity that all children, all students have to drive the process organically.

That seems to me like a future of education that is infinitely better than what we currently have. And it would be unlocked by AI that understands our minds.

Okay. Well, we've got a very wide-ranging conversation. I can tell that you're a podcaster, which, no, it's a good thing. I mean, it's like some people who are, you know, some people like are just like not, they're like sort of maybe like camera shy or they don't know how to sort of light up on a conversation.

So I would send everyone to your podcast and to check out my conversations. Any other, you know, we talked about Jason a little bit. Any other places that typically people should start at in terms of like getting deeper into your work?

Well, with the podcast specifically, season one, which was kind of popping the hood on the lab and talking to diverse folks, but season two is going to be very different. I'm talking to many different scientists, social scientists, Amazon scholars, really anybody who has something to say about what a mind is and how we can go about building one.

So it's going to be a very different set of thinkers, also industry insiders. Yeah. I'm really excited. I'm having the conversationsright now and they are very different from season one. Maybe I'll get you on.

I don't know how to make my own mind sometimes, you know, so that's one of those things. I'd be excited to listen as well. So thank you so much for the time. I'm really excited to work with Amazon AGI for the World's Fair.

You guys have, you know, like a really good presence that is coming out and I think like really kind of appropriate for emerging as a lab that I'm excited to see. Like I think you guys have been, you know, you specifically, you know, even through a debt, been working on this for so long and like, like I want it to happen.

There's so much drudgery and knowledge work that I'm caught up in that like, oh man, it's 2026. Like how come it's not solved yet? You know, that I'm trying to like string together individual tools that I don't really work.

A very simple one. Okay. Like we're recording on Riversideright now and I got to get this on YouTube and the whole process of download and edit and upload and all these things. There's like three humans that touch this.

Yes.

And I'm like, why?

Right. I mean, yes, that is a problem that we are trying to solve.

And like humans are great, but also they're slow and expensive. But like, you know, I have to, we have to align and that's the stuff that will never go away. Like I need to be like, no, we're going to cut that.

We're going to emphasize that. We're going to pull this out and focus on this. And that is the part of the editing that I think, you know, I have a particular type of knowledge work, but everyone has some kind of knowledge work that looks like that, that like you're interacting with systems, but then also you're that systems are interacting with you.

You're interacting with other humans that are involved in the process. Let's solve like, you know, I'm excited for a future where we solve that, but that's not even what you're driving at. You're driving at like how do we like we build entire civilizations?

How do we bring out the next generation? I think very exciting.

It starts with the digital drudgery though.

Starts with the digital. Yeah. Right. Yeah. I mean, well, that will immediately fund everything else. Like you will get all the monies

because no one wants to do it. It is so funny. Like I have been very excited about, you know, computer use and perception agents and RPA for a long time, but it's not there yet, you know, and I think I can see progress in this decade.

It's like, it is it. Like we're sort of living in that last period of time that was kind of maybe created by software that we're now sort of creating agents for software so that the problems that software created are also solved by more software.

At a certain point, we have to stop thinking about software.

Yeah. We're not there yet.

Cool. Thank you so much, Danielle. I hope to see you back in San Francisco and I'm sure we'll chat more.

Yeah. Thank you so much for your time and your fascinating questions.