OpenAI o1 isn’t a chat model (and that’s the point)

Hey, everyone. Welcome to the Latent Space Podcast. This is Alessio, partner and CTO at Decibel Partners, and today we have a special episode. We have actually a podcast that is preceded by an essay that Ben Hylak wrote, um, on Latent Space recently about how he was wrong about o1 and how to actually prompt it to, to get good results.

And then Dan McAteer has been a friend of Latent Space and part of the community for a long time, who also did a very nice video on how he automates some of the prompting for, for o1. Uh, what we wanna do today is twofold.

So one, how do you go back and tell yourself that you're wrong, Ben? Maybe we can start from there on, on o1. And how do you get Sam Altman to retweet you and say, uh, many such cases? Um, and then we can kind of go into the, the specifics of o1.

But, uh, welcome to the podcast.

Yeah, thanks for having us. Yeah, we, we were talking about this a bit before, and yeah, I think it's, I think it's, I think it's actually one thing that people that are, like, deep in this space don't get about people that are outside this space.

So I think that, you know, we're-- a lot of us are engineers, right? So I think we're used to being on the cutting edge, and we're, like, excited to try stuff. Uh, at least myself, I'm excited to keep trying stuff 'cause, like, I want, uh, I wanted o1 to be good even when it wasn't.

At least, like, when-- the first time I tried it, it wasn't good, but I wanted it to be good, so that's, that's kind of why I kept trying. I think most people, that's not true, right? Uh, like most people try something once, and if it doesn't work, they, like, they have this, like, very fixed mental model.

They never try it again. So there's definitely that just kind of engineering mentality. And then I think separately, like, uh, you know, we work, like, uh, my company works with a lot of AI products, and so A, like, we have to have good opinions about this sort of stuff, and, uh, we also, like, when we're wrong, we'll hear it from our customers just chatting with them, right?

So it's like a lot of it was like, you know, I have this tweet and like one of our customers is like, "Hey, actually, I like it," you know, and here's what we use it like, you know, we're planning on using it like this in our product, et cetera.

So that feedback loop is really-

Um, what was your experience then? Did you like the model right away? Like did you kind of, uh, you know, did it grow on you?

Yeah. So I think my experience with o1, where it's been different from other models for me, is that o1 is actually the first model where I'm getting more impressed by it the more I use it. So like when ChatGPT first came out, right, I think it was GPT 3.5, and at first it seemed like, oh wow, this is amazing.

It can actually create text that sounds like a human, and it can write poems. But kind of as you started to use it more, you would discover more of the things that where, you know, there was error cases or there was areas where it couldn't, it wasn't capable.

And with, with o1, it's been, it's been the opposite experience. So the more I've used it, the more impressed I become by it and the more I realize it can do. So it's kind of like by the nature of it is so capable, it's sort of been like pulling me towards using it more.

Are there for either of you any use cases that you were not able to do at all in previous models that you didn't even try at first in o1, and then you found out it was actually good? Any lessons there?

I think this is, this is interesting. Like, uh, one, one thing you'll see a lot and like even reactions to the post is like people looking for specific examples where, you know, one is a lot better. I think Dan actually did a great job of like pulling out some of those examples.

I think it's really hard because a lot of it is just like deeply specific to like, like I think that it generally understands the project I'm working on better, for example, um, and like the intricacies of how the code base is laid out.

There's like we use ClickHouse a ton. Um, I think specifically it seems to like confuse like quick, uh, Click, ClickHouse, uh, SQL, um, syntax less, but these are very hard things to like articulate to people in an example.

It's like, oh, uh, you know, it's like, uh, sixty percent, seventy percent better or something, which is, which is huge.

I would love to just run through the post because I'm sure people have read it, but we can get the behind the scenes of writing it. So, um, I think the title is the most important thing, like o1 is not a chat model, and I think this is both from a usage perspective, but also ties back to some of the training stuff and post-training that we already talked about.

Like the previous models were so focused on early chat based on especially on chat completion that they're great at chat, and they know what to ask next. The o1 models are more goal and reward based on the, on the post-training side, so that kind of changes how you interact.

Um, what-- when did you first make the distinction? You know, is there kind of like some day that you were using the model that you were like, "Okay, like I just need to completely change the mental model to think about it," or was it just kind of like, you know, a gradual progression?

There, there, there actually was a day and it, it, it's, uh, it's kind of funny. Uh, I think that, um, I j- I was just trying to do something over and over again, and it's obviously super long response times.

Uh, separately, uh, there was-- I was having a bug with the ChatGPT app on my phone where like it wasn't, uh, like every time I like went to a different app, it would like stop responding the, the o1 response, which was like kind of silly.

So I was hitting all these bugs, and so the only way to get around this bug was to just like not have to iterate with it, right? Like, essentially like, uh, I, I was like, I can't keep doing these back and forth 'cause every one takes like, you know, five or like, you know, five minutes to respond, but then every time I would accidentally like do something, it was, you know, adding more time.

So then I actually took the time, like not thinking about it from like a, oh, you know, like model perspective, whatever, but just because I was tired of waiting so long between, uh, iterations. And, and then the, the answer was really, really good.

Um, so that's when it, that's when it clicked for me.

Did, did you have a similar experience then? I'm always curious, like I didn't really think through the chat. It's almost like the chat is still the UX, you know, and I think that's what confuses people maybe. Like if you didn't have a chat interface and it was like write a brief, you know, I think people would maybe be steered more the, the right way.

So I'm curious then what was your path to, to kind of work through that?

Yeah. So what I'm, what I'm mostly doing when I'm using it is I'm using the, the connection feature in the ChatGPT app to connect it to Cursor or to Windsurf or to VS Code. So typically what I'm doing is it is still the chat interface, but I'm really thinking about it as like an intelligence layer over my code editor, my IDE.

So and I think for me, like at first-- So I'm typically using LLMs mostly for, for coding use cases and like using Sonnet 3.5 and GPT-4o, I would, I would often get like ninety-five percent of the way there with in, say, implementing a complex feature.

But it's kind of like in programming, if you're ninety-five percent of the way there, you're zero percent of the way. Like you need to get a hundred percent of the way there or it just doesn't work. Yeah, using o1 was, it was the first time where I would connect it to my IDE, I would provide the full context of my code base.

Like I'll just create a file that concatenates all my files into one just simple text file, give it to o1 and then say, "Hey, based on this code base, help me, help me implement this feature." And it was the first time where like it could do it one shot and get me a hundred percent of the way there.

So from there I was kinda, I was kinda hooked just because it works so well.

Um, yeah, Manuel, aka Slono from our Discord, he built a tool for like a, a text file merging, um, for code use cases. Has this new kind of like prompting pattern changed the way you think about prompt storage?

Or like do you keep a doc with like some of these like, you know, long context just to like re-prompt it for like the same type of task or?

Yeah, I just keep a, I just keep a directory in my pro- in my repo, my projects, and I just have like o1 prompt template in there. So like this weekend I put, uh, Ben's structure in there, and I'll just use the coding agent, and I'll just say, "Hey, create me a prompt based on this input using this prompt template," and it can just generate it for you.

Ben, you wanna run through the prompt template?

Yeah. Um, well, I don't have it in front of me right now.

Well, I have it. I'll read it out for you. So the, the anatomy of a o1 prompt, so you have the, the goal, the return format, warnings, and then context dump, which if you think about putting a ton of stuff in there, you should put 10x-

Yeah

... whatever you think a ton is. How did that structure come to be, and maybe what are like the, the things that people get the, the most wrong in putting it together?

Yeah. Yeah. I, I think the big thing about, and even like maybe something that I saw some people misinterpret, was like what I found that worked really well was not telling the model how to think about it, if that makes any sense.

Like almost more like you're actually interfacing with a person, which is like scary in its own right. But you know, if you're, if you're working with a, uh, like a coworker, you're not gonna be like, "Think really like slowly about this," or think, you know, like it's, it's kind of patronizing.

But you might say like, "Look out for X, look out for Y. There's this weird thing in the code base," right? Like you're gonna give those sort of warnings. I think that it's trained to reason a specific way, and it's, it's best to sort of let it do that.

I think that similarly, even things like return format goal, it's like similar. Like instead of trying to, uh, prompt it too much on how to think, I'll kind of make sure that the return format itself lends itself to good results.

So I kind of used this example, like made up an example around like hiking. Like for example, you'll see it says, you know, "Return the name like you would see it on all trails," right? And it's like kind of trying to get it in the direction to like not just make up a random name, like actually try to make it.

But it, it's a little bit less than like, you know, telling it not to hallucinate. Like, so I think for some reason focusing really on that re-return format also is, is really critical.

Are there bad return formats that you tried and kinda moved off of? Or, you know, like the hiking example is one. Are there any other good or bad examples that come to mind bo- for both of you?

I mean, interestingly, like I think, you know, uh, uh, the hiking is somewhat like a convoluted example, 'cause again, I think it is sort of hard to have an example that actually like is sort of universally applicable to people, which is like what my goal for that.

Um, like I think that if I, if I was actually looking for hiking things, well, I actually did try that prompt, and I, I think that something with like access to the internet actually will do better in those cases, which makes sense.

I think that I talked a- about it a little bit there. I think that I've had a very hard time getting it to actually write stuff. I know that I've heard of people using it for writing where it's like processing diffs, more like providing critiques or feedback.

But at least for myself, I haven't found a good way to get it to write in a specific tone or like match my tone, for example, or really match anyone's tone. But if there is a way to do that, I would, I would love to know.

Yeah, for me, at first I was asking it to generate full files of code, like as I'm making changes. That can get a little bit confusing, and then if you're trying to use like a coding agent to bring it over into your projects.

So I feel like that's, that's not the best format for coding use cases. And so what I started to do, which, which Ben alluded to, I started asking for the return for- format to just be a diff, so a diff only.

So I only want you to give me back what I'm going to change in my files.

Yeah. Um, yeah, I think that's an interesting-- It's kinda like blurring the line. The second point that you made on the post is describe what you want, not how you want it. Um, and I think there's almost this like gap between, "Well, this is what I want, but I kinda want it as a code diff," you know?

Um, what I...

Yes.

Yeah. I, I'm curious, like how, how do you think about describing, like how much of an importance should how you want it be? I mean, going back to the tone, right? It's almost like people are so focused on, "Oh, I need it to be my tone," but then maybe if it can write a more compelling copy and it's not in your tone, then that's just as good.

Yeah.

Right? So how do you think about that?

That's interesting. I think the tone one for specifically, I think, I mean, even taking a step back, I think the sort of tough part here is that, um- Even with LLMs in general, I think that, uh, you know, e-even since like, you know, GPT-4, GPT-3.5, like there hasn't been an instruction manual.

Um, I think that, you know, even OpenAI doesn't really have an instruction manual. Like they sort of like this thing is, is born and they're trying to figure it out in the same ways that we're trying to figure it out.

Uh, I think that what makes o1 even trickier than other models is that, um, there, there is actually an asymmetric-ness to how well OpenAI understands the model and how well we-- For example, the fact that like reasoning tokens are hidden, right?

So there's all this stuff happening that we actually just can't see, right? Even as developers, which is super interesting. Um, and then also there's this whole component of like how it was trained that we're also not privy to, which is like, I think very different it seems like than how other sort of like what the industry standard is.

So yeah, just, just prefacing that, I think that the tone thing is somewhat a side effect of the fact that it has all these reasoning tokens that is in a very specific like tone. It feels like very academic tone.

So I think that's where that comes from. I do think that the describing what you want part is extremely, extremely important. So for example, in, in, uh, in, in Dan's example, I think that if you're trying to sort of like make changes to a production code base or like something that you're going to be maintaining, for example, you probably do want diffs.

Like you wanna make sure that you understand exactly what it's changing and asking for a diff almost a way of signaling to the model that like you don't want it to just go change a bunch of stuff. Like you want it to make sort of very measured, targeted changes.

Whereas like you can imagine that if you were making like just some sort of app you wanted to spin up in a like in two hours, that like having it just YOLO every single file, you know, and like just throw everything into one big file, like that could be totally fine.

So I do think that that like process of really critically, so exactly what Dan was alluding, that process of like really critically thinking like what is the exact, what is my intent of asking this question? Like, what do I, what am I trying to get out of this?

It feels like more important than ever as models become more capable.

Sorry, I misclicked the button. Um, I wanted to ask Ben in your actual day job, which is running Dawn Analytics, it's almost like if you're building a product that does analytics and observability into some of these new models, how should people think about that?

How do you evaluate some of these models? Like how do you understand how people should improve it, especially when you have kind of like this long tail of use case where maybe people are trying to jam new models into old use cases.

Are there any heuristics that you think about also, like, I should not be using o1 for this? Or are you just trying to upgrade most of the tasks and workflows to, to be o1 compatible, so to speak?

Yeah, that's really interesting. I think one really fascinating thing for us where, you know, for, for folks that don't know, we're, we're essentially like a Sentry for AI products, where we help people catch issues in production. And, uh, I think that models are actually getting harder to use, which is like very interesting place for us to be, where, you know, we're, we're finding that like o1 is the most capable model I think that, uh, OpenAI has made, and it's also, I think, the hardest, uh, to use.

And so being able to understand like how users, uh, are using it wrong, being able to catch those issues are the ones that you're, you're not seeing actually, I think is becoming more and more important than ever. I think that, um, as far as like how applications should be thinking about using it, I think what I'm excited about is it being used in-- I think that we, we could see real like background level intelligence.

I think this has been a promise like, uh, an example of what I mean by background level intelligence is like you can imagine that you have something looking through your GitHub repo and it is like finding, you know, oh, you should actually be refactoring a certain part of your project a certain way.

I think that that's something that like, you know, you could have, uh, the quality of answers continues to scale with compute time essentially, or like tokens outputted. That actually all starts to get very exciting. So that's, you know, that's a coding example, but you can imagine this across every single app, just like actually having...

That sort of promise, like AI insights into your data could actually become real. Like I think in a lot of ways that there's not a huge delta right now. Like essentially if you're using like GPT-4o or something to like do those sort of insights, it's like, okay, really like thirty seconds or sixty seconds of compute, uh, no matter what.

I think it's going to be really tricky to figure out how to nail user interfaces for this as far as like using o1 as a direct result of a user's input. I don't think OpenAI has done it yet.

I think that, uh, it's like the, the number of tasks that you're willing to wait, you know, like three to five minutes for is like probably pretty similar to the number of tasks you're, you're willing to wait like an hour for, which is, which is interesting, right?

Um, so, uh, yeah, I'm not exactly sure how to reason about that yet.

Was that the same for you then? For, uh, I mean, for coding I use Sonnet 3.6 unofficial name. I get pretty good results most of the time, but it's all in the composer, kinda like a UX. What, what have you switched over to, to o1 that you couldn't do before?

What are you still doing with the old models instead?

Yeah, I think anything where it just seems simple and like you can do one-off and you don't need to bring a ton of context into it, you can, you can typically use like Sonnet or, or GPT-4o for that.

Anything where I feel like if I was going to try to implement it myself and I would need to pull in context from a bunch of different files and it's maybe connecting to several different features or parts of functionality, that's where like where there would be that cognitive overhead, I'll just give it to o1 to let o1 do the, the thinking on that.

And we just got-

Hello. Can you hear me well?

Somehow well, yeah. What, what well is it? Yeah.

As well as I can manage. Uh, yeah, I was caught in lunch. This, uh, this is very hastily scheduled, but time is of the essence, right? Yeah, like we, we use, uh, I think, uh, we-- Ben, you and I talked a little bit about how we use o1 for AI news.

Yeah, very like it's a very long context use case, lots of instructions. Creating the brief is kind of what I, what I call it. I definitely editorialize that on your article, but the more I do PRDs, the more I do design briefs If I feel like I'm talking to o1 when I'm talking to humans, and I'm taking that same approach to my o1 usage.

I'm curious, like we sort of touched on this even as we were writing the article, but, um, you mentioned using it for news. One thing we were just talking about earlier was like, I have a very hard time ever trying to get it to write in a specific tone, like actually output something in a different tone.

I'm, I'm curious if you've found a way around that, if you fi- have a similar experience or sort of h- exactly like where you use it for, for writing a- as far as th-that process.

Yeah. We, we talked about this a little bit. I showed the screenshot at the end of, in the essay. But so one, the tone that you're trying to do is kind of more bloggy, your voice informal. Whereas the tone that I'm trying to do happens to be the default of OpenAI anyway, so we're not that far off.

Two, I do have examples that I captured from its previous output that were bad, and I do have examples that were good. And what I do is actually I ask, um, 4o's perfectly fine at this, ask it to diff, ask it to self-critique.

I'm like, "Here's, here's the good example, here's the bad example. What do you think you did wrong?" And they're like, "Oh, like, you know, I see, I get it. Like, you're more specific here. You like the name dropped here.

You don't like this thing that I do." And so then you copy and paste that critique back into the prompts, and it's gone. And actually, I found that that is a better way of doing this than when doing, you know, XML bracket, good example, close bracket, bad example, close bracket.

Those, those examples tend to leak. There's an e- there's an issue with prompts leaking, example leaking, where the, the, the few short examples that you put in the prompts actually show up in your output, and you don't want that.

But you can use the critique of the diff to actually influence the, the style. Uh, and that is my very, very amateur soft prompt. And what you actually want is representation fine-tuning or, or soft prompting or any of those o- other more advanced things.

But when you're just normal APIs, this is all you can do.

Yeah. You know what, what's fascinating and o- one thing about as you're talking is like, it feels like if you, even if you look at the last, like, ten years of, like, ML and AI, there's this sort of like, uh, you know, uh, divergence and c- convergence of, like, models where you have a bunch of little models and then one model that does something re- like, kind of converges it all, and then, like, a bunch of little models.

Um, or maybe a bunch of separate steps is another way of saying it. Um, and it feels like we're d- like going back into a time where having these separate steps with almost, like, varying levels of intelligence becomes increasingly important, like this idea of chaining.

I think for some amount of time, you know, when 3, 3.5 Sonnet is, like, both pretty fast and pretty intelligent, and I think I could do a lot of things in one, one sort of prompt. I think Flash sort of like became so much faster that I would then start using kind of both.

But it feels like with, you know, how, how good o1 is and how long it takes, and then, yeah, I, I just find myself going towards having multiple steps with, like, smaller models checking the outputs a lot more often as well.

Yeah. Dan, you got any, any, uh, thoughts on, on this stuff? Unless you guys already talked about it.

Yeah, well, one thing I, I was thinking about too is like, so I'm at a, I'm at a startup. We have a, a chatbot that you can deploy through Teams or Slack, right? And it's, it's a compound system, so it, it uses that model that Ben was talking about where you have like some small models that will do certain tasks, and we, we haven't implemented o1 at all into the system yet.

I'm sure we're thinking about it. In those use cases where latency is a, is a question, that's definitely something to think about. And what we've done in some of these cases is we'll create a, like a sec- a separate module where you can type in a command and you could, for example, like enter reasoning mode.

But I think still the challenge is like you need to think about when you need to use the model in your head to think about when is it relevant to go into reasoning mode and use the more powerful and more expensive, slower models.

Yeah. I, I sort of touched on this a little bit in the article as well, but I definitely think that, you know, we sort of had a chat around like I, I had this line in there like, you know, o1 doesn't know, uh, it can't reason about how much to reason.

And it's like that's not really true because it does sort of have different lengths, right? But it does actually feel like if you give o1 a very simple question and you don't give enough context, like it'll generate, it'll, it'll try to think really hard about it, uh, at least sometimes and like come up with this sort of like ridiculous answer.

And so I think, uh, point, I think it's actually simple things are often better, you know, like really, really simple tasks, especially if you're not gonna provide a lot of context, like it's often better to just use a, a smaller model for a lot of reasons.

Yeah. Um, so, uh, I, uh, that w- I, I found very few issues w-with your post except for that one, and I was like, I, I seized on it and I was like, "No, like the community's gonna have an issue if I just publish this as is."

So, so yeah, I mean, I, I, um, I edited that one. There is gonna be a trend this year. People are always asking me like what trends they're expecting to see. You just reminded me, model routing, you know, is coming into vogue a little bit.

Basically, what we're doing here is that like you're talking about having the human, the user as the model router. Um, and eventually, like we're going to need the models to be model routers to, to understand their own traffic.

Um, there's obviously some, some startups doing this. The Martian is, is one, and then LM Systems also have a model routing project. I think there's gonna be more. Anthropic and OpenAI have both teased their model routing on, um, APIs already, and so I think they'll launch them officially at some point where you can sort of prioritize the three trade-offs that are in model routing: cost, speed, intelligence.

And, um, it makes, that makes a ton of sense. So it's, it's gonna happen. Like now humans are doing it, but eventually they won't. And I think it'll initially do a bad job and eventually get better. That's how, that's how these things are.

Yeah.

Well-

I think that like understanding intent is like the, the holy grail of a lot of, a lot of these use cases and, yeah, it's just super fascinating to, to, to think about how it's gonna be done, both from like an architectural level, but also like a sort of interface level.

Like I think that one of the most frustrating things for users is like when if something starts, you know, doing something, like I'm thinking back to like when ChatGPT initially launched the internet search and they were sort of trying to be very smart About how and when that happened.

Um, and it was often super frustrating, right? Like you were just asked, every question you would ask, like vaguely could, you could imagine it could be used for internet search like it was searching. But it, it got a lot better, right?

And like it, um, they sort of found a way to tone that, you know, find a good balance, so.

I mean, the, the, the balance was you have a button. There's a search button. You turn it on-

Actually, right, yeah, yeah. They originally sort of made it very explicit.

They gave up.

You're right. But it, it became a button actually. You're right. That's correct.

But Canvas, Canvas is grounded. So they, they try to be smart about whether they're using Canvas. Still not great. I, I, I say I always, when I, when I want Canvas, I say, "Use Canvas." When I don't want Canvas, I, I don't.

They, they have a UI way around it. Okay. O-one more thing I, I wanted to cover. I don't know if you guys already covered this, but, uh, coding slides. Dan, you, you had a video which I think you, you showed off, uh, some tips on, on how, how you were doing stuff.

And then like, uh, Ben, I felt like a lot of your examples were driven from copy-pasting code into ChatGPT. Is that how you code? You don't use Cursor like, you know, like, like a normal person?

No. So I, I go back and forth. So honestly, I do use Cursor. Um, I actually just recently switched to Cursor, believe it or not. I, I've actually been sharing some of my thoughts. Like, uh, switched probably, uh, like a month ago or so.

So like after, after the sort of initial hype. For context, like I had, um... When we first started our company, we had like a super early code gen plugin in VS Code called Sidekick. So we had like shut it down publicly, but we kept like adding stuff to it, and like, honestly, it's really good.

So we, we kept using it for a long time. I think for specifically, uh, a lot of what I've been using it for is like, uh, SQL queries and like generating like a lot of like back end and architecture stuff.

So that, that's where I, I do use ChatGPT. I have found like differences in... So for example, I actually generally get better results out of Claude in, in, you know, just claude.com, uh, versus, uh, Cursor a lot of times, um, which is interesting.

Like I love being able to apply code, et cetera. But as far as just like a lot of times I find it getting stuck in sort of a loop that I don't see in claude.com and, and so I, I haven't seen that for, I haven't tried it too much for o1 specifically.

Uh, but, uh, that was just sort of an experience I had, uh, early using Cursor, so.

Alessio, Dan, up to you.

On the coding side, I don't use, I always use the Composer Cursor interface, so 3.6 is, is pretty good, but maybe it's just skill issues. I used to do the o1 planning and then Sonnet to like implement the, the changes.

But I actually found that like there's not that much of a difference. But I should re, I should try and redo everything with o1. Whatever from all the founders I work with, it's mostly they have great success with o1 as kind of like a, um, outsourced person.

You know? That's most of what it feels like. Like one of them, they were trying to get this like, uh, marketing positioning document done from an agency. They were charging like 30 grand, and they were just not getting the right results.

And then in like about five, six hours, like the founder created like a 20-page doc through, with o1 Pro, uh, just like pasting things and like explaining, basically giving the same direction it would give to the, to the agency, uh, but instead doing it through a model.

So, um, I know we always joke about 2025 is definitely the year of the agents because everybody said so, but, um, I, I think it's gonna be interesting, um, to draw the line, you know? I think maybe 18 months ago you would've said this is a marketing agent, you know, that you're kind of, that is kind of doing things for you.

Now, uh, we compressed that into a model. So, um, I'm curious to see on the coding side. I, I don't know what, for example, what, um, Devin is doing, like as far as, uh, where you draw the line between chaining, just using o1 and like the environment is kind of what matters.

So I think a lot of questions are open on, on that side. But I think, yeah, people underestimate how much we can do with a good, a good brief like, uh, like Ben suggested.

Yeah, so one of, one of the criticisms that I saw, and I don't know how many of these comments were just bots, I'm sure it's a non-zero amount, but a lot of people were like, "Oh, you're really gonna write out that much?

That seems like so much to write for just a single prompt." Which obviously that was not the point of the post. The point of the post was to illustrate like how you can use a structure. But I mean, I think it's, it is a valid criticism if you're trying to write out a prompt like the one that Ben suggested by hand every time.

That is, that is a lot of work and a lot of typing. So what I do, and I, I think what you should do, is use other prompts to sort of bootstrap your, your prompts. So like I have it set up in Cursor or in Windsurf where I was telling Alessio before we started, you basically, I have like a prompt template in a prompts directory, and I'll just say, "Hey, Windsurf, create a prompt," and I'll just dump a brainstorm of an idea, and then now it has Ben's structure.

So it will use the str- Ben's template, the structure of that, like o1 prompt, and it will generate for me just based on my like high level thoughts, my brainstorm, like a fleshed out prompt that follows that structure.

Cool. I know we only got a couple minutes left. Any other thoughts or maybe questions that you have on these models, like, uh, open areas of, you know, tinkering that people should do that you want to, to have them report back to you?

I mean, um, yeah, there, there were a couple interesting discussions. I, I think one, one thing that came up was like a lot of questions around context before, context after, which is like actually super interesting. Claude, you know, famously does super well with putting context before.

Um, they, like Anthropic mentions that in their guides. I haven't seen a huge difference trying it either way with o1. Like in, in some parts of their documentation, OpenAI recommends putting it after for prompt caching. So like one would assume if they recommend that, they're also like training, you know, um, in that direction.

Uh, but I would love to see more evals on that, on that topic specifically.

Yeah, I, I think the, the practical concerns trump evals.

Mm-hmm. Yeah.

If you're, if you're trying to take advantage of prompt caching, then you're not actually changing the context. So that means you don't have a ton of context. Anyone who's doing, who's working with a ton of context has to put them at the end because they're swapping them out.

That's just how it is. I don't see any way around it.

For me, it's like taking it, the first thing that came to mind is taking it in a direction slightly different from coding and engineering. One of the things that I've like kind of tinkered with it for, which I need to do a little bit more, is trying to find novel and unique connections, like let's say, um, like across scientific research.

So like give it maybe 10 different scientific papers from 10 different fields as context and say, "Find me opportunities for trajectories to take different areas of this research that maybe no one has thought of." So like try to find these unique, uh, research areas that you could go into that are things that people haven't thought of before.

I think that, I think it's, it's hard for one person to take in so much data, so much information and like make connections, and I think that's definitely one of the areas where these models, they're probably already superhuman at doing that.

Yeah, we, we haven't yet seen someone, uh, stick o1 in a room for like two or three days, uh, by itself with its own thoughts sort of thing. Uh, it would probably be like thousands of dollars, uh, uh, which is, which is one of the things that is honestly even just scary, like, uh, uh, experimenting with it, right?

Is it's like you want to try something sort of silly, it's like that's $20 every single time. But, but yeah, I would love to see that as well.

Cool, guys. Thanks for joining, and I'm sure we're gonna have an update when o3 is made available and see how, how the model evolves. Um, awesome, everyone. Thanks for the, the time and, um, see you all later in space.

See you all. Bye.

Thank you.