The antidote to AI fatigue — Answer.ai Solveit

Hey, everyone. Welcome back to the Latent Space Podcast. This is Alessio, founder of Kernel Labs, and today I'm joined by the Answer.ai gang, Jeremy, Eric, Johno. Welcome, guys.

Thank you. Hello, hello.

Thank you very much. Nice to be here.

Jeremy's on number four now. He, he gonna be basically like a, a co-host at this point. You've been obviously a, a fan favorite. Um, Eric and Johno... Johno, I think a lot of people met you at different events.

I know you were at ICLR this year. You were in Europe a couple years ago in New Orleans. Eric, obviously people, you know, have known you. I don't think there's many people in the Bay Area, or at least that are in the startup world, that have not heard about you.

So we will not do the usual formal introductions, but, um, E-Eric and Jeremy, you guys started Answer.ai now almost two years ago. I remember we were doing legal for the term sheet over Thanksgiving.

Mm-hmm. That's right.

I was in Wisconsin, and they didn't understand that people worked after 5:00 PM. Let, let's give people an update on how you got here, and we'll talk about some of the exciting stuff you work with with Solveit and...

But I think there's maybe a question of why did it make sense to build Solveit, you know? And so let's start, let's do the quick, why did you start Answer, which we did a, you know, a good amount of content in the past.

And then what was the idea maze that got you to Solveit, and what is Solveit a tool for in your mind?

Yeah, start, Eric. That's it.

Gosh, I can't believe we're two years in already. Yeah, you know, Jeremy and I have been talking about what to do with AI for a long time and how it is that we could maximize the probability that AI is used for useful stuff.

And we just felt like there were a series of research and product directions that weren't being explored in the gold rush over, over AI and the, and the eagerness of people to get to AGI. And we, you know, had this idea that we could, we could build a really different kind of AI company.

As I think we talked about in the last, uh, in the last podcast, you know, we were inspired very much by the Edison Labs and the advent of electricity, where... Which, which is like a bit of a throwback to an older time where research and development were not seen as two independent separate fields, but rather one integrated process.

And we wanted to see, like, could we really push the envelope? What kind of team could we build if we took the thesis of AI's transformative potential seriously? And we both know, and I know you do too, Alessio, many, many, many AI labs whose rhetoric about the power and transformative potential of AI is unmatched, but whose actual behavior is relatively conventional.

It's still organized as a relatively conventional company using relatively conventional techniques. And we thought that doesn't really make sense. We ought to be able to see what is possible if we treat AI tooling as a kind of a, a ability to create superhumanly productive people.

How many, how many of those people would you really need in order to create something really transformational? And then, of course, we also really believe in, you know, the MVP iterative development approach. So rather than try to build one massive super application, our idea was to try to, uh, in parallel, explore many different possible domains where there was, uh, a new way of, you know, finding something useful to do with AI.

And, you know, we could talk a lot about the philosophy of it and how it led to Solveit, but, like, very broadly speaking, the idea is to have... really do AI with human in the loop, not just as like a check to make sure the AI is doing the right thing, but really the AI-- that the human is the agent driving the process end to end.

And we have found that, that basic idea of breaking problems down into sub-problems, really making sure that as the human operator, you understand each of the sep-steps and sub-steps of the thing that you're doing, has allowed us to be just ridiculously productive in a wide range of applications, which I'm sure we'll, we'll talk about.

And yeah, so that's, you know... So we're two years in. I can still remember exactly where I was. You know, I remember the term sheet negotiation we were having over Thanksgiving. Like, I, I remember certain seminal moments, and I can remember where I was when Jeremy told me, by the way, he doesn't think the company will ever need more than 12 employees.

And I was like: "Oh, sure, you mean like this month," you know? But he's like: "No, over its entire life." And I think that's also been really transformational, is just by, by creating an artificial constraint at the beginning, forcing ourselves to really try to find truly exceptional multidisciplinary people who can go deep into many, many, many different topics, who are, who are true generalists and who really believe in the mission of using AI in the way that Jeremy has, has advocated for, this, like, humanistic approach to AI.

I've just been shocked at how much we've been able to get done. Even as someone who spends a lot of time around very, very productive startups, this is at a whole different level.

I mean, to give a sense of that, we've... this group of, which has averaged about 10 to 12 people, currently nine, I think, have built a pretty transformational and complex, uh, piece of software, which we can do a quick demo of later if you're interested, using a complete web application development platform we built in-house, uh, using a styling system we built in the house, deployed to a new deployment platform we built in-house, doing all the DevOps and sysadmin and everything using our own tools that we built in-house.

No AWS, no Google Cloud, whatever. We've built our own fully integrated professional services firm providing legal, accounting, bookkeeping. They're all using Solveit, by the way. They're all using our AI. And we were their first client, so all of our back office and everything is being done by a complete system developed in-house.

You know, that's pretty unusual for a company, I think, with thousands of people that's operated for some decades, and so it is a really cool level of productivity to have.

Yeah. Uh, you know, I was... I interviewed, um, Dylan Field from Figma yesterday.

Oh, great.

And one thing we were talking about is, like, how do you have software as, like, one, a way to limit the projection of your ideas into the latent space, you know? So, like, how do you, like, take away a lot of the mental complexity?

And then on the other way, it's like there's so much into the model that, like, it's, it's almost overwhelming to get it all back, you know? And so how do you build these experiences? And in their case, it's like, how do you explore, you know, different designs and, like, what's possible and what comes back and what goes in?

And in your case, it's almost like if there's, you know- Billions of economic value in the latent space of the model is, like, what's the tool through which you expand that back from, you know, training is compression, and so what you're doing is almost like expansion, right?

You're, like, trying to turn that back into, like, valuable things. So yeah. How did you get to this approach? Were there, like, other interesting approaches that you tried, uh, that didn't work? Like, what were maybe the core tenets?

So I think just, you know, this is the first time we're having the in-depth conversation, but based on what I've seen, one is step-by-step explainability that is, like, branchable. You know? So there's kinda, like, the main task, and then there's, like, explaining and understanding each piece of it.

Was that there from the beginning, or is that something that you came up after? And then, you know, the handoff between AI and human when it comes to, like, writing code, for example, and like, uh, you know, I'm sure also, Eric, you've been writing your book using it, so even though it's not code, you're still writing some sort of artifact.

Yeah, maybe you talk people through the different patterns that you've been through.

Yeah. So Alessio, it's, it's funny. It turns out that Eric and I have both spent some decades building up the same ideas and trying to convince other people of them in quite different areas. But the idea in both cases is basically do things in as small iterative steps as you can.

Get as immediate and accurate feedback as you can. Go on to the next step. And Eric studied this as a concept much more than I have and documented it in his previous books. You know, and it, and it does go back to the, the OODA loop of, you know, aerial dogfighting and the Toyota Production System from decades ago and so forth.

It's certainly not something actually that we've come up with from scratch, or at least, like, it's... We're kind of rediscovering things other people have discovered. So I've spent years trying to embed that idea into software development. And again, I wouldn't say I'm doing anything that new there.

Stephen Wolfram certainly has been doing that for decades with Mathematica. So this idea of doing it at every level of the organization and everything we do, so, so when our lawyers now work on contracts, they're doing that same thing, small steps, you know.

And what was the interesting discovery here was that when we built a platform that allows an AI to see all of those steps and to contribute to those steps, you get all of those benefits of lean startup, Uber, you know, uh, OODA, then B Dev, whatever, multiplied by the capabilities of an LLM.

And then you add agent pieces. So one of the really interesting things we've done is we've made it easier to add tools to the LLM than anything exists in the world. Literally any Python function that you put right there in that system is immediately a tool.

And it's extraordinary what happens when you write a three-line function that can call Grep or RG or whatever, and then you say, like, "Hey, LLM, you now have access to this tool." You know, go figure out why this PyPi module I downloaded is broken, and suddenly it zips off.

So yeah, I think the idea is basically an expression of, um, what we had both been working on for years. Does that sound right, Eric?

Yeah. And I also would say that there's a bit of a return to, like, the general purpose computer in this that I didn't even really appreciate. I... A lot of people of our generation, you know, we started out programming computers, like, when the dominant platform was, like, a PC that was relatively hackable.

So it was like the idea that a, that a computer is something that you can kind of program and reprogram was a very natural part of what it means to be a computer. And as we've moved both to more consumer level devices that are not...

They're, they're closed systems. Also, as we've moved infrastructure into the cloud, we now... Very often, we are interfacing with general purpose computers mediated through layers upon layers upon layers of interface, which sometimes are d-designed for composability and reusability, but quite often really aren't.

They're about having a defined workflow and keeping everybody in their lane, and I, I, I... And I don't mean to criticize those products. They've been extremely helpful and useful in a lot of situations. But it's been really interesting because, because having access to LLM in a kind of a native way makes a lot of the safeguards and kind of like handholding that those systems provide less needed.

It's been really interesting to just combine, you know, like this kind of LLM style intelligence with the general purpose computer that's just always on and available to you and mix them together, and it's kind of... To me, it's felt a little bit like hacking felt like in the early days of learning to program.

There's just been so many times when, you know, I wanna solve a problem, and as you say, I've, I've lately been mostly using the product to write prose, not code. But even, like, to have a bunch of data from a research report, you know, in a CSV file and be able to just, like, instead of having to go somewhere else, figure out what to do with that, think about it, bring it back, and just, just add that to the canvas directly, be able to access it programmatically and then to pull that information directly into the writing process.

It's just created a much more natural flow for a really wide range of tasks that I have found very surprising, very interesting.

Just to explain the, the general purpose computer thing.

You know, one of the things we really tried with Solveit is to make it as easy to use as ChatGPT. And so you can just, you can just start a chat, you can start chatting. But behind the scenes, we've actually launched a persistent Linux Docker container for you and attached it to a unique URL for you.

And so you actually now have your own computer. It's basically a VPS that you're, that you're now running this in, and so you can install software and create servers. And so, for example, one of our team created this really cool thing called Discord Buddy, which is ...

kind of like take some of the things we learned from Solveit and puts it into Discord. Um, can show you later if you're interested. That's actually running. It's a server run... It's just running in a Solveit instance. It never got deployed anywhere.

And so this is this really cool thing where we're, we're, we're giving folks this ability to fully harness a computer that they, they can instantly spin up and learn about with the help of AI, and with the help of experiments, and with the help of iterations, and they're not limited by what SaaS APIs you have, and you don't have sudden horrifying bills because accidentally you ran a big query thing over five terabytes of data and it ran it o- across every row or something.

It's, it's all, you know, it's affordable and-

Very transparent

... and convenient, and it's more like having a 1990s shell account. This might be a bit before your time, Alessio, you know, where you upload a PHP's file to it and instantly anybody in the world can start playing with your program.

Yeah. Well, yeah, at, at my age, it was already FTP. You would get FileZilla and like some server and then just dump things into it.

There you go.

Um, yeah, maybe let's just do a... Yeah, let's start from a demo. I think like give people-

Okay, sure

... a kind of tour and just what it looks like, and then we can kind of talk through the different pieces of it.

So, yeah, I mean, basically,

when you start, you can just create a chat or load up an existing chat. We call them dialogues. And you'll remember, Alessio, we, we told you guys about dialogue engineering before this idea of context engineering existed, so that was actually first discussed, uh, on your podcast.

So dialogue engineering actually goes a lot-

As usual. As usual.

It actually goes a lot further than what most people think of as context engineering. But yeah, I can start typing, um, "Hi, can you tell me something about the latent space?"

And, uh, you know, we're, we're chatting. And so it's... And it's gone and done some searches, so these links are gonna be, you know, to more information. Look at that, Jeremy Howard, Chris Lattner.

Yeah.

That's a great bunch of guests.

I know that guy.

Cool.

That's a, that's a good duo right there.

Chris Lattner.

Yeah, pretty impressive lineup.

Who's he? Um, so you know, like you can treat it just like ChatGPT.

And so, like how should people think about these though? Like do you call them chats? Do you call them-

These are dialogues

... dialogues? Like-

Yeah. These are dialogues

... yeah, dialogue is the name of each.

Yeah, ex- exactly right.

So basically like a conversation plus the VPS plus the tools make a dialogue.

Yeah. So this is, this is a Discord... This is a Docker instance running here. Um, and so I can, I can name this. So this is my latent space.

If I, um... You know, you can see I've got a, this kind of unique URL that's been created, and if I, uh, go over there, you'll see... There you go. I've got a running latent space. So Kernel is the thing behind the scenes that's running the dialogue.

And, you know, where it gets interesting is, um, you know, I can, I can change the, the answers even to, to redirect things any way I want, right? So for example, here, I could change this and say, "Did you mean the actor or the data scientist?"

And now I'm actually changing answer.

Um, and this is very important because, as I'm sure you know, Alessio, the, the auto-regressive nature of language models means that if they make a mistake and you correct it and then say, "No, that was a mistake. Please do it this way instead,"

the more often you do that, the worse the dialogue answers get because it's... In the training data, when it sees mistakes, it tends to be followed by more mistakes. So this way, you can continually guide it by actually a- changing its answers, right?

And this is one of the pieces with a lot of existing chat things that drives me a little crazy now that I've seen the difference, like the fact that you can't go back and remove messages, shift the order, change the responses, edit the history.

Your only options are like, "Oh, this context is getting polluted. I have to start a brand-new clean slate." Yeah, that very immediately feels like such a waste to throw all of that useful context away just because you've gone off the rails on one piece.

Yeah. And then you can, um, add, um, notes, which is markdown messages. So here's our, um, you know, about the podcast section. And so then I could add a, a coding in Solveit section.

And so just like you can treat it like it's ChatGPT, you could also treat it like it's a Jupyter Notebook. So instead of being in prompt mode, I'll switch it to code mode, and I'll say, uh, you know, print, I'll say like A equals one, B equals zero, A over B.

Um, that's like... And now a key thing about this dialogue and a key fundamental thing of like basically all the AI work we do now is we realized the AI must be able to see everything that we can see exactly like, like we see it, and vice versa.

We need to be able to see everything that The AI can see. So here I can say like, "Damn, I broke something. Please tell me what went wrong here." And one of the other unique things we do is we have these modes you can choose, and in learning mode, it's gonna do everything it can to try to make sure that I'm solving my own problems.

It's teaching me things in really small steps. It's using all the best information we've learned about how to, how to teach the best. And we can also ask it to use thinking mode, so it's gonna use reasoning. Um, so it's having a good old think there.

And so you can see here, it's not going ahead and giving me pages and pages of code to say like, "Here's everything you need to know," but it's like very small steps. "Would you like to see how to handle this situation in your code?"

What models are you guys using for this?

We're using Anthropic models. Um, but we're not just using system prompts. We've got a whole kind of different system for how we get it to work the way we want it to. And actually... Actually, here's a, here's an interesting thing now.

By having it go in these small steps and being, you know, we, we... It gives us bits of code, and we can now immediately run that code. So I can hit a button, and it pulls the code out, you see.

So by kind of having this integrated prompts and code all in the one place, I can immediately run it, and then I can say like, oh, it's very interactive, you can see.

So we've, we've seen a lot of folks using this. Actually, I'll give you a sense. It's like it... So we, we kind of had the, had this test run, Allessio, where we, we thought like, "Oh, this has been so useful for us.

I suspect other people are gonna like this as well." So we decided to just let a few people try it, and we opened up, along with a course, this kind of preview a year ago. And we opened it up, and we said, "Okay, does anybody else wanna try this idea?

And we're gonna teach you a bit about it." And within 24 hours, we had 1,000 signups, so we immediately shut it down. And so we just did 1,000 people a year ago. And we actually have... It's amazing, we have this, like, list now of hundreds and hundreds and hundreds and hundreds of bits of feedback from those first thousand people.

You know, everything from a professor of English to a college student to a psychiatrist to a 15 years of experience AI engineer. Um, and I think they, they're interesting to analyze to see what people got out of it, right?

So the professor of English noted, "It provides a hedge against AI tools that totally replace the human creator. Comes out of a deep philosophy of AI that fits with a deep commitment to craft and attention to detail," which I...

Hopefully you kind of get, right? This idea that you do things in small steps and you're trying to learn from it. You're trying to do small pieces. Um, you know, this guy's an EVP, so he's kind of helping realize he can use this to help the developers in organization.

And in fact, uh, one of the alumni kind of created a showcase of projects that were developed during the course by students. Again, this is from just the number of people that signed up within 24 hours. And in fact, he built this whole page using Solveit.

So he's got this description-

Oh, nice

... here of how he built the whole thing using the stuff that he learned in our course.

Yeah. So that's a very quick, um-

Yeah

... overview of what's going on.

Do you feel this... You know, there's a lot of talk about, you know, more user-generated software and kinda like everything moving more towards personal tools to help you versus, like, building these kinda like SaaS tools that try to help everybody, and then maybe it doesn't quite fit what you do.

How do you think about that? It's like, you know, if I had Solveit, like, you know, Eric probably is not gonna buy another word processing thing for writers, you know? He might just keep using Solveit, and then, you know, you do this for learning Python, and then you kinda play out.

I know, Johno, you love plants, and so I'm sure there's gonna be a smaller mar- even smaller market now for, like, plant analysis apps, you know, because you can just kinda do it yourself. Is that kinda the feeling that people should have?

Because I feel like, in a way, the, the name almost is so broad. It's like Solveit. It's like it is like the it, all, all of it, you know? How, how do you think about what the, people should do with it today and maybe, like, the vision that you have for it overall as, like, the impact that it should have on people's work and life that is not work?

I'll take it because I, I know, uh, people... I, I get asked quite frequently about how I use AI. You know, it's like a very common question I get from other writers and also from founders. You know, "Are you using it?

Are you not using it?" And I've actually had to be a little bit careful what I say because as, as a writer, it's made me really productive. But as soon as I tell people that, they're like, "Oh my God, you're letting the AI write for you?"

You know, it's really scary. It's like, "No. Oh, no. You cannot let the AI write for you, okay?" Just super disclaimer, you cannot let the AI. You will wind up with slop just like everybody else winds up with that.

That is not helpful. But you don't appreciate until you have the right affordances around you, like, how much crap there is in almost every workflow to do-

Would it help to show, Eric, to give a sense of how you-

Oh, do you wanna see a demo? Yeah. Happy, happy to show like, um-

And I wanna make sure we got very into it here from Johno as well 'cause, uh-

Yeah, yeah

... he, you know, I think he's got interesting ideas about this.

Probably, maybe Johno's demo would be better to do next if you wanna do a demo.

Yeah, sure.

Uh, I mean, I was just thinking about the, you know, plants, you know, Johno. I'd love to hear your thoughts about, about what, about Allessio's question and then how you analyze sets-

Yeah. Let me just-

... and do with plants

... let me answer the question, sir, just because you're... The... W- what it's done for me is it has changed my perspective on what is an app in a, in like a very profound way because, like I-- there's like a huge range of problems that I would never, even if an app for solving that problem existed, I would never feel like it's worth it to download an app, build, download app, install, w- integrate into my workflow.

It's like, look, I need to use the default tools. I'm too busy. I can't be learning a new thing, and it's just like I, I have a very narrow view of it. And yet, just by building up really bit by bit what I've needed just for the writing tasks that I have, I now have basically a completely custom piece of software that is highly tuned to the kinds of writing tasks that I've been doing with the new book.

So Johno, maybe you can give some, some demos of just, like how these applications come together, because I think it changes our idea about what is an application in the first place.

Definitely. And Alessio, I thought it was funny that you said that, like one of the tabs that was open but that I wasn't planning on showing was- ... feeding some pictures of plants to AI as a, a sort of mini eval.

Yeah, I guess you're right. The, the, the sort of just-in-time software spinning up custom pieces, it's... Solveit and, and AI more broadly, it just feels like such a potent piece for people. So this was a thing. There's a, a fumery, a, a perfume shop I walked past, and I just read Gwern's post about avant-garde perfume, right?

So if you wanted to l-learn if there's a perfume that smells like wet asphalt, I wanted to make a little, like, search for the, the scents. And, um, like, obviously, uh, perfume people will be yelling at their screens because these are like text descriptions of, like, very ephemeral pieces.

Um, but I can show you what it took to make that, and it's... We're not talking about like a multi-week, you know, multi-person endeavor. It's like an hour or so of fiddling around. I, I, I did this as a, a demo, kind of role-playing someone who's a bit more of a beginner.

Um, but you'll see, like I'm in this document, this dialogue. I'm laying out like this is kind of what I wanna do. I wanna get some smells and embed them and search them. I'm picking up some data, and then the AI here is very much like it's in, it's in the learning mode.

It's kind of the default we leave it in, and it's only ever suggesting very little pieces. Like, oh, here's how you might get rid of all of the excess HTML. Okay, Beautiful Soup, that's a fine idea. Um, and so then I'm, if I'm on this message, I hit that key.

I have the code that the AI suggests, and I can edit it and try and-

Can I just mention, the reason for this is, like, we wanna make sure at the end of building something, two things have happened. The first is you understand every line of code fully, why it's there, what it does, and that you've learnt from the process, so at the end of this process, you're a better developer or sent analyst or whatever than you were when you started, which is very different to vibe coding, which is...

maybe also has its place, where it's like, "No, I just wanna get the artifact out as fast as possible. It's a one-off and I'm done," you know? Whereas most of the work we do, we wanna improve our capabilities as human beings, you know, and, and build an artifact that we can con-continue to work on.

Yeah. And, um, it's also like this, this task is, is small enough that you could almost do this... I think you could give this to an agent today, and it could quite competently build you a, a smell search engine, except that there's pieces like...

So here, for example, I gave it the, um, I gave it the, the docs from Gemini. Uh, here's the, the Google Gemini embeddings info, just so that it had that in context, and then I asked it for the code to, to get the embeddings of some text, and it gave me the snippet, which I split up and ran.

Um, but then you'll see, like it's calling, um... It wants to show result.embedding zero, and there's an error. And so you could imagine, like I ship this off to an agent, it, it does all of that, and then at the end of the whole process, when it's written everything, you try and run it, and you get the error, and now you have to do the loop of, "Okay, can we fix this?"

Whereas here, because everything's live, you can see what does result look like. Oh, it's actually something that has a .values where the actual numbers are. So I know the, the, like, collaboration of I'm in this environment, the AI's in this environment, when there's a-an error that crops up, you can inspect the output immediately.

I can see how to fix it. If I wanted instead, like I could just ask the AI, you know.

And the AI, and here, like instead of like in Claude Code or Codex and store something, it's generally gonna have to guess at what went wrong. Whereas here, you or it can immediately add a, a message to check exactly what's there in this dynamic environment and immediately fix it, so it's a very efficient software engineering process.

And how... Sorry, how far can the branches, so to speak, go? So could I respond to the system there and kind of just keep traversing down a conversation-

Uh-huh

... system, fixing the error-

Mm-hmm

... and then go back to the main?

Yeah.

I'm just curious, like how you would think about that.

Yeah. You can dump it all in a collapsible heading section, for example, and, you know, kind of go down rabbit holes or delete them when you're done or keep them. There's a lot of flexibility.

Yeah. And, and curating the context, I know people are, are coming around to this idea now that that's really important. But, but this takes, takes that context curation to a whole different level. And one of the things that it's made me very, very aware of, just as a user of the tool, is how the attention mechanism works.

Like, you, you start to get good at directing the full model's attention to the thing you really want it to be focused on in the moment of what you're working on and, and avoiding dis- avoiding it getting distracted.

And what's... It's just so interesting to me how similar that is to human distraction. You know, like no matter how intelligent you are, if you spread that intelligence over too wide of a surface area, it's really hard to focus.

It's hard to, to make progress. And so it's like-

This interface thing is cool too, Eric.

Yeah, go ahead.

I think we should definitely get Johno to show this.

Yeah. So, um, like Alessio, you, you could see I, I kind of just collapse away my, my rabbit hole, or I, you know, can delete those messages if it's too far of a, a thing. But it, it makes it really, um, effortless.

I think the goal is to encourage people like, "Oh, okay, I need this, this measurement. What is cosine similarity? If I want, I can go and have that chat." Um, I don't have to start a separate chat and load in, "Hey, I'm working on this problem and I would like to learn about cosine similarity."

Like it's all there and ready, and then you can scrap it all or keep it there for reference, depending on what you want. Um, and then, yeah, I think the bit Jeremy likes, so here I wanted to make the actual little search widget.

And this part is something that I, I'm actually very familiar with, but I also imagine, like, is very tangential to I would like to search embeddings. I just want to get some UI up. And so I told it like, "Hey, go figure out how to show the results of this search in a nice way."

Um, and then it designs a little card, and I get the, the live preview of exactly what that looks like.

So you can see why here, where, Alyssa, you're having this REPL kind of browser is another unique kind of capability. You know, you can build a web app or a report or a graph or whatever interactively in the back and forth, the answers.

It's all there, right? So he's creating a web component, and he's now got a clickable, usable web component right there.

Yeah. So it's, it's all, it's all live. It's very easy to share this result. Like, this is just at a, a URL route, and there's a public URL for all the Solveit instances. But yeah, just having the, like, back and forth of, again, you're not starting a new context to say, "Now let me build a UI for this," and not knowing what this is and having to explain it.

It can see we're doing search, we're getting five results, so we should probably make a search route that returns the results, and we have our search function that we've already written. So all of the pieces, even though by this stage in the dialogue, I was kind of just, like, relying on the AI to do the actual writing of the pieces of code, there's still pieces of code that very much fit in my head and can be immediately tested, right?

I can immediately preview that and then make a new page and, and open it in the full size app. There's no, um, skipping out to some new context. So the, the fact that it all fits in my head and it also all fits in the AI's head because this dialogue is the context that it's seeing, you know, rabbit holes and all, if I want to include those, um, from the start where we've got the, like, initial, quote-unquote, prompt for me and the AI, this is what we wanna do.

We're gonna embed some things and then make an interface. So it's very satisfying to work with. Yeah.

Because it's your own... Yeah, because it's your own virtual server that's running, you don't have to deploy it somewhere. It's like literally the thing that's running inside that dialogue. You can view it and interact with it. And so, like, our, our legal team, for example, do a very similar thing with, with contracts, where they have, you know, they're going paragraph by paragraph.

The AI can see exactly which paragraph you're currently reading. You can refer... You know, you can say like, "Hey, what if we did this?" or, "What if we did that?" Or like, "Can you check out the Delaware Chancery records to see how they would respond to this?"

And then gradually build up, you know, this, this other kind of artifact, which is a contract or, you know. And of course, in Eric's case, I mean, one of the interesting ones for Eric is he's been going through all this feedback from his trial readers in a similar way, right, Eric?

Yeah. Yeah, yeah. Um, what I was just also thinking about, um, fact-checking. You know, fact-checking a long document is still well outside the capabilities of most AI systems because, you know, for the attention reasons. But if you're working in sections, it's really easy to be like, "While I'm here, just do you mind doing a quick fact check, make sure everything in here is, is accurate?"

So I'm happy to demo that if you want, along with the test reader comments, if that's okay.

Yeah. Yeah, yeah. That would be fun.

God only knows what... Hold on. Let me make sure I have the right tab. Give me one second.

Don't spoil the ending.

Well, I'm like, I'm like, "What is actually gonna come up here?" Okay. So, all right. So here, can you see that? All right. So I am working on chapter 11. So hi, everyone. I'm, I'm working on a new book.

It's not announced yet, so don't, don't look too care- too closely at what this is, but the book is called Incorruptible. It'll come out next year. And I have some kind of system prompt like stuff, like here's a list of...

This is a Wikipedia list of AIEs things to avoid, 'cause I found test readers are exceptionally sensitive to AIEs now. Can't use an em dash for anything, or people get really upset. Here's like a briefing about what all the different chapters are about and, you know, just to give you some context about the book.

And then the rest of this is just, um, you know, just code, like Jeremy was saying. So here, he uses some meta programming to figure out what chapter it is that we're working on, so that it can find the, the name of the file where that chapter is, is stored.

I, I work in Markdown. And we tried to do this in Google Docs, and their API is so bad that it was actually easier for us to just make our own Google Docs replacement in Markdown to do this in.

So anyway, so that's, uh, a lot of this is just boilerplate stuff relating to managing the context. But here I can ask it, okay, this is just a general purpose thing. Here, the actual chapter is now loaded into a variable here inside of Solveit.

Give me an evaluation of it.

And the AI can see those variables.

Yeah, it can see these variables. So you look on the side here, you can see that TXT is a variable that contains the text of chapter 11. Um, I'll skip over its detailed analysis of my style and tone issues that I gotta address.

Um, work from a to-do list. And then this is a new thing I've just been playing around with. I use this test reading platform. Shout out to helpthisbook.com. I've had a few test readers going through and reading the manuscript, and they can flag with comments things that they found confusing or interesting or whatever.

So, uh, this is all me giving it instruction about what I'd like it to do. But if I come down here, I will show you. I'll run this. This is a little bit of code that I, that Solveit helped me write to find...

Oops, the old one. Find all the comments in the CSV that relates to this, um, chapter, and it will give me a little summary here. I got 4,489 comments I gotta work through, of which 168 of them relate to this chapter.

And then I can be like, "Great, uh, give me an analysis of these comments. You know, what do people think about them?" And so you can see here, this is a tool call. It has gone to grab all the comments that relate, that filter related to this particular chapter.

And here we go. Here are things that people really like. They like these seven things. They found these things confusing. People have requested these. Here are some specific comments from specific test readers that relate to recent changes. So I get a nice overview.

But the nice thing about it is, now that I go through, I'm gonna go through step by step by step by step, section by section in this chapter And we're gonna do them one at a time, and it's gonna follow the same procedure each time, which is to find the relevant section of the text, find the relevant comments that relate to that text, and I'm using Solveit meta programming, which we haven't talked about, but a very easy way for it to, like, create its own messages to create a briefing note, which is just a single message that has all of the context related to this section of the chapter.

So here's the text of it. Here's its assessment of things that I need to work on. Here are some relevant comments that relate specifically to that section. And then, you know, I can say, "I wanna do the next one," and it'll, like, gonna go through and do this section by section.

And here, this one, I was working on this just before we started. I was giving it some correction, like, "Oh, we need to make sure that, um, this stuff needs to happen." It's like, "Oh, okay. Well, I'll update the briefing note then to make these fixes."

So then it went back, and it actually updated this text. You'll notice my four points are right here, have been added to the note and, and off we go. So then we can check for redundancy with the chapter.

I can say, you know, "Give me a web search fact check of this section."

And if you give it a second, it will do that. And you know, just like any AI tool, you have to make sure that you don't let it turn your brain off. That really is, like, to me, the number one thing I've learned from this and all forms of vibe coding and all the things that I've, I've, uh, played with, is that it's just extremely easy to get into this kind of like soporific state where the AI is doing the thinking for you, and you start to think that all the information that it's generating for you is like an asset rather than a liability that you're getting...

Having trouble with. So anyway, so this-- Yeah, the, yeah, yeah. And like, I happen to know this fact is in fact true, but I understand why I had trouble finding it. Anyway, so you can't, you can't just, you can't just, uh, rely on what it says.

But it just-- I don't know, for me, being able to transform this, like, very complicated task of, "I gotta work on this book today, I gotta make progress on this chapter, I gotta achieve this goal," and just break it down into these extremely discrete points of like, "Here's one reader comment.

They were confused about this word in this section. What do you wanna do about it?" It's just, it's been a game changer.

Yeah. I mean, to me, that's almost, like, more impressive than the coding use case because I think you're, like, eliciting something out of the model that you, you just could not in any other interface, you know? I feel like if I squint at, like, the coding use cases, like, well, if you have a Jupyter Notebook in IDE, you can maybe...

It's still not as effective, but you can kinda... Like this, it's just, like, impossible. Like you just-

No. Nothing, there's nothing like it

... simply-

Nothing like it.

Yeah. You just could not do it. You cannot put the whole thing in chat, in Cursor and ask it to fix it. So that's super interesting. How do you guys feel about composability of, like, the blocks themselves? I feel like some of those tools that you had in that dialogue, Eric, you might probably wanna use in a future book, or like other book writers-

Yeah. Yeah, yeah

... might be helpful in using the same, the same system. How do you think about, well, maybe it's like a dialogue library? I don't know.

It's, it-- I, I, I thought this was gonna be super complicated and difficult to do when we first started building this out. Yeah, dialogue libraries and templates and things that would be shared and, and all credit to Jeremy really for this.

By, by getting back to the general purpose computer, it's like, it's actually quite easy. I skipped over this, but in that dialogue you'll see that it said something like import reader comments as a, as a Python module. I have a special module for reading these comments out of a CSV.

That is actually just a different dialogue in my, in my Solveit. It's named reader_comments. Jeremy and I paired for 25 minutes or an hour. I don't know, w- for a few minutes one day to be like, really think through what's the best way to put this, get, make this into a Python module.

It was more like 25 minutes. Yeah.

Yeah. It was like 20 minutes. We just, it was like we were just riffing one day. "Oh, you know, let's, let's mess around with this." I didn't, it wasn't like I was planning to bring reader comments into Solveit.

It just occurred to me one day, "Oh, I should do that." We did the dialogue, and I remember we got to the end and, you know, and, and Jeremy was, was helping me understand how to use the dialogue to accomplish this task, not just doing it for me.

And we were kind of pair programming together and asking Solveit to explain the things it didn't understand and figure it out. And when we got to the end, I remember saying, "Okay, now, um, what do I need to do to, you know, publish this module or this dialogue?"

And he's like, "You, you don't have to do anything. It's already done. That file is now an importable Python module on your, on the disk of this general purpose computer that you're in. So go into your other dialogue, type import reader comments, and you're done."

And that was really mind-bending to me because it's making the promise of composability, like, possible in ways that it really wasn't before. And now we've started to do more fancy things, like we have some modules where after you import the Python module, it kicks out a bunch of notes with extra context information.

We have this idea of, of turning functions into tool calls, so it automatically will make some of the Pythons, it will promote them into tools that you can call. So it, it gets a little fancier than just regular Python import.

But you can start to see how, you know, this, because it's just a file on disk. So then when we started to talk about multi-user Solveit, it's like, well, let's have a shared mount and put the file in the shared mount, and then you can import it in both places.

Like it's, it's just, I don't know, somehow programming has gotten away from these really simple abstractions for, for natural reasons, but it's been really fun to, to get back into that. And so just to close it out on the writing thing.

So I, I, like, effectively have a highly customized Eric's exact writing system that I like to use. It's perfectly the way I like it. And it's not a piece of software that I wrote. I don't think of it that way.

It's just a dialogue that every time I wanna make a new one, I just duplicate the one from before, change the title to be the chapter that I'm working on now, and rerun it. And I delete the old stuff, and I customize it a little bit each time to, to deal with the task at hand.

But it's like a really interesting question of like, so what is the application? People ask me, "What do you use for writing on AI?" And then one answer is I just, I use Solveit. You should use Solveit, too.

If you could access it, you, you could try it. I feel really guilty every time I tell people that. Like, we should really probably make this available to you, too. But another way I, I'm using an application that doesn't exist for anybody else in the world.

It's my completely custom thing. It wasn't vibe coded for me. I wrote it myself, and I understand every line of it. But also, I had a lot of help in making it. It was actually very relatively easy to make.

It wasn't like I said, "Okay, I'm gonna spend a month writing this piece of software and ship it." I mean, last, I remember last time I wrote a m- a Mac application, you know, I just spent a month like, how do you make a DMG file again?

And how do you sign it? How do you publish it? And just like the amount of time I spent on the UI of that application compared to how much time I spent on the meta programming of like how to make it into an application was probably like 10 to one the bad way.

And now, and if you ask me, "Would you write me an application?" If ever like a friend comes to me and say, "Hey, do you mind writing me an application real quick to solve problem X?" My mind is not thinking about how easy is solving problem X.

My first question is like, "How do I en- encapsulate this thing into a form they can access? How much support are they gonna need when the dependencies change? Who are they gonna call when it doesn't work?" And it's just like, forget it.

The answer is no. I'm not gonna write that program for you. I'm sorry. Even though the actual thing is as easy as generating the Fibonacci sequence, you know, probably would take me five seconds to write the actual logic for the thing.

And I think that's really a really interesting promise of Solveit, that we could get to a situation where those caliber of applications can be written economically, because all the questions are just-

And to be clear-

Yeah

... we made Solveit for ourselves entirely, and for you, Alessio, our investor. We had to create something where we could be the platform that would allow 12 people to create thousands of valuable products and, and investigate what AI is good for.

Yeah. How do you think about the picking? So you have, you know, Eric built it himself. Um, what are the things that you wanna guide people to build or dialogue through, and what are things that you wanna build kinda like first party, so to speak?

Like, what's that decision-making process?

We just do it. You know, like we really, like we just, we just do it. Like it's like a, like Johno wanted to have wife searching sense. He just did it. What, what happens is, as we scratch our own itches...

Actually, can I show you something? This, I think this is gonna blow your mind. Um, we discover things that just work great, and we wanna make available to everybody. So I'll show you something we do wanna make available to everybody, although we haven't yet.

Now, how do I do this? I think it's here. So we discovered, for example, that, and this is something we do all the time, is

we, we don't just have meeting notes. When we have a meeting, we have, we ever do it, we want it to be incredibly valuable, and so we wanna create something out of it that, that's really rich. And as we did that, we realized like, wow, our...

You know, Eric's seen every meeting notes AI app in the world, you know?

Of course.

And he was like, "How did you make this? This is so much better than any other meeting notes. It's so valuable." It's like, "I just did it in Solveit." And so we are like, this is such a great way to turn, um, discussions into,

into text that we took on Andrej Karpathy's challenge, and his challenge was, "Hey, can anybody take my Let's Build the GPT Tokenizer video and turn it into a blog?" And here is that blog we haven't released yet. But look, it's got pictures.

It's got all of his code, tables,

you know, hyperlinks.

I mean, this sounds like a great thing to put into Solveit to understand, right? Is that the-

Yeah

... the idea to-

Yeah. And you can... Exactly. You can load this into Solveit and actually run this code and try it out yourself. But, so we took on this challenge, which, you know, nobody else has managed to do yet, which is to kind of use AI to create a very, very high quality version of a very, very high quality video.

And basically, yeah, you know, this is something we, we're kind of building an API around this. We wanna make this available not just in Solveit, but for anybody to turn their discussions, their videos, their interviews into very, very high quality text outputs.

And the way we do it, again, it's not send it off to an AI to do it. It's, it's have a just discussion. So here's the actual dialogue. So, um, and one of our guys, Kerem, did, did this.

As you can see, he, he's going through it line by line, you know, paragraph by paragraph, starting with the transcript

and saying, you know, um, "Okay, here's the bit of the transcript," right? "Go ahead and show me the code." And it's like, "Okay, here's the code." Um, and it's like, "Okay, add a hyperlink to all of the Python functions mentioned."

Right? So when this mentions or, there should be a hyperlink. It's like, "Great. Okay, add the docstring. Okay, great. Here it is." And so this is way faster than writing out that whole thing yourself, right? But it's also way slower than just saying like, "Press a single button."

But again, it's, it's this human in the loop going through. And the nice thing is that with the dialogue engineering we discussed, the longer your dialogue is, the better the AI gets, which is the opposite to what we're used to, right?

Because you can edit the outputs that aren't great. So it's like, "Oh, this, this is nice, but I need some minimal code examples for UTF comparisons

So yeah, that's like a,

an example of, of something that we've started out as being we're using this for all of our own transcripts and turning things into notes and discovering it was working really well, and now we're like, okay, we wanna turn that into a kind of boxed up application for everybody where they can take advantage of this, and they can also do it in Solve It.

Awesome. What should people expect from you guys, I guess, like on a basis?

So we've got... Yeah, so we've got our first full version of the Solve It course coming up. You know, it won't just be a test preview. We're taking everything we've learnt from the last year, and we're gonna make it available to many, many more people.

So folks can, if people wanna try this, they should sign up. So, um, the URL is solve.it.com for Solve It. So come to solve.it.com, check that out. There's a really great Discord community of people who have now for a year been help- all those thousand people have been helping each other out for the last year.

You know, at l- hundreds of them have told us it's changed their lives, and now they're helping each other change their lives. So it's the most kind, helpful, powerful community of people I've ever seen. And in fact, Eric, you remember when, when we first talked about it, I said like, "Do you think we should make this available to other people?"

You said to me, "Yes, the number one thing you're gonna get out of it is a community," right?

That part has been really exceptional.

Remember that I'm a Italian citizen, so if you need solve.it, I can, I can use my-

Please

... Italian-

We'd love solve.it

Yeah, yeah, please. Yes, use your, use your influence. Absolutely.

We didn't manage to get that one, but, you know, the .com is pretty good.

That's true. That's true. I should... I, I got a lot, a lot of new friends in Italy. I should, I should have asked about this. Um, Jeremy, do you wanna give the dates just, um, in case people are listening and they wanna know when to sign up?

'Cause last time it, it... sign up p- period wound up being quite short. That may happen again, so we might wanna just let people know-

Yeah, I mean, we wanna work, leave it open for longer this time. Like we've, we've, we've, we've got- we've built our own horizontal scaling deployment platform, Alessio, so we can handle a lot more people now. But it is quite a lot of work to host these full computers for everybody's chats, as you can imagine.

So it won't be an infinite number of people that we can handle. Johno, do you have the dates offhand? It's October the 20th, I think, is when we're gonna be starting.

The 1st, I think, I th- is what it says.

Yeah, I think we're gonna start the 20th.

We'll open signups earlier than that.

Yeah.

We'll open signups probably beginning of October.

Well, open, well, hopefully by the time people watch this, signups will be ready.

Right. Yeah.

So you should be able to come and sign up right now.

And then the course is gonna be-

It'll be five weeks

... I think like five weeks, several lessons every week, but also live demos and explorations. The, the funnest part by far is seeing what everyone else builds with the tools. You know, it's-

Definitely

... always surprising and exciting.

Yeah. There'll be office hours.

A lot of interaction. Yeah.

There'll be... Yeah. It like, some folks last time kind of pretty much were like full-time doing the course. Like it's a... You get as much out of it as you wanna put into it, basically.

We hired some of them, right?

We did hire some of them, yes.

Some of our best employees. Yeah, that's true.

It's a great way to hire these exceptional people, actually.

It's been really great. Alessio, I don't know if you want to include in the show notes, we can send you some of the blog posts from people in co- in cohort one. I feel like that's actually more powerful even than any demo.

We'll throw everything in there. Um, awesome, guys. Any other call to action for people? Even, you know, requests for research, I don't know, anything that, that you wanna get out to the world.

I would like to say that you don't have to do this approach in our tool. Like I, I like the stuff that we've done, but a more general appreciation for this iterative, like human-driven way of working. It's one of those very paradoxical things that can feel slower than the magical tools of the day in the short term and is extremely rewarding.

So if you've been getting into coding or if you're building tools, like try it out. Like spend some time attempting to understand the pieces as they come together, um, and doing things in small chunks and, and learning, and it's really like...

It, it feels good. Come on in. The, the water's warm.

Yeah. Yeah. It's actually a really important point that maybe we didn't emphasize enough in this conversation, is that the, the course is really about how to think in the Solve It way. The tool is just one possible implementation of how that way can be, can be used.

So I'd actually encourage people who are, who are thinking about it, um, we're gonna do a unit on writing with Solve It, not, not just with code. Also, for people who have not learned to code but would like to, this actually is a very good way to learn programming.

I say as someone who's taught a lot of people programming in my life, this is a really exceptional way to do that, and we're also gonna do a unit on entrepreneurship. So if you wanna think about, I feel like one of the big missing pieces in all this AI revolution, all these AI agents, is like, where are, where is the new software?

Where, where are the MVPs? Like, there's just not as many MVPs being created as there ought to be if the software works as well as it claims it does. So we'd like to actually really help people turn these into useful applications.

That's our, that's our mission.

I should mention, yeah, talking of missions-

Happy to help

... Alessio sup- and his company supported us in becoming a public benefit corporation. Our job is not just to give Alessio the highest possible returns. Our job is actually to maximize human flourishing by taking advantage of AI. Eric's book is gonna be all about helping companies deliver actual value to society, and he'll be talking during the course about how to, you in your company, or if you want to start a company, can do that.

So what Johno said about like, "Hey, you don't have to use our tool," you know. That, that's very... That's our values, right, is we're trying to... We wanna help everybody flourish, even if that's not necessarily the profit maximizing step for us.

Yeah. No, that's definitely been from day one, I think, the most important thing and the thesis behind why Answer should exist, which is why... Which is more people should have access to this in a way that makes them great and makes them flourish versus us making the most money.

Because at some point you don't need more money. You know, you need more, more people to do great in life. So yeah, thank you guys for taking the time, and I'm sure you'll be back on the show soon.

So looking forward to that.

Well, and thank you for all your support in these past years.

Yeah. Thank you, Alessio.

Really great to partner with you.