# The Accidental AI Canvas - with Steve Ruiz of tldraw Latent Space · 2024-01-05 Steve Ruiz, founder of tldraw, recounts how his open-source whiteboard accidentally became a premier multimodal AI canvas, enabling visual prompting with GPT-4V. Originally built as a personal project with obsessive attention to arrows and ink, tldraw gained traction as a developer SDK for infinite canvas apps. The 'Make It Real' feature, which turns wireframes into working HTML using GPT-4V, exploded with 22 million Twitter views, showcasing iterative design through spatial branching and annotation. Ruiz explains how the canvas supports multiple input layers—UI sketches, state charts, screenshots, even API endpoints—and how his fine arts background guided his taste-driven approach. He argues tldraw's moat lies in its web-native, hackable canvas infrastructure, not ephemeral AI integrations, and positions it as the battlefield for comparing multimodal models like OpenAI's and Gemini. ## Questions this episode answers ### Why is tldraw considered an accidental AI canvas for multimodal prompting? Steve Ruiz explains that the infinite canvas lets users dump screenshots, text, images, and annotations, turning it into a powerful environment for GPT-4V. Unlike linear tools like Midjourney, the canvas shows progression, allows visual branching, and enables iterative refinement by drawing on previous outputs, creating a more natural design loop. [43:07](https://addtry.com/870358af-50ec-4dfc-87cf-18eaca243c6c?t=2587000) ### What was tldraw's Make It Real feature and how many views did it get? Make It Real lets users sketch a wireframe and instantly get a working HTML prototype from GPT-4V. Steve Ruiz says his team expanded on a Figma developer's demo, and the feature exploded, receiving 22 million views in 30 days on Twitter, which he calls 'Kanye West numbers.' [22:06](https://addtry.com/870358af-50ec-4dfc-87cf-18eaca243c6c?t=1326000) ### How much does tldraw's SDK reduce the time to add a whiteboard to an app? According to Steve Ruiz, the tldraw SDK shrinks the development time for embedding a whiteboard from 'three years and three people to three weeks and one person.' He notes that many teams, even those not building whiteboard products, can now integrate an interactive canvas with minimal effort. [17:22](https://addtry.com/870358af-50ec-4dfc-87cf-18eaca243c6c?t=1042000) ### How did Steve Ruiz's fine arts background shape his approach to building tldraw? Steve Ruiz, who holds an MFA in visual art, says studio art trained him to follow genuine interest without external constraints. He applies this as a founder, prioritizing what feels most compelling to work on, which he believes leads to better outcomes and is part of tldraw's 'secret sauce.' [1:23:24](https://addtry.com/870358af-50ec-4dfc-87cf-18eaca243c6c?t=5004000) ## Key moments - **[0:00] Intro** - **[0:59] Background** - [1:13] Steve Ruiz's background is in fine art, holding a Master's from University of Chicago, entering tech at 30. - [6:27] Steve Ruiz created 'Perfect Arrows' to precisely solve the aesthetic challenge of drawing good-looking arrows between boxes. - [7:40] Perfect Freehand: an open-source library for variable-width vector ink, solving a problem no one had cracked before. - [10:06] Perfect Freehand is now used by Canva, draw.io, Excalidraw, and tldraw, becoming the standard for digital ink. - **[10:54] Tldraw Origin** - [14:09] A large communications company paid $75,000 for early source access to tldraw, funding its initial six-month development. - **[15:44] Growth** - [17:22] Tldraw reduces whiteboard-like feature development from 3 years and 3 people to 3 weeks and 1 person. - [20:06] Tldraw amassed nearly $200,000 in total sponsorships before raising a seed round from Lux and Amplify. - **[21:37] Make It Real** - [21:40] A Figma developer created the viral 'Make It Real' demo by using tldraw to transform wireframes into working websites via GPT-4V. - [22:44] 'Make It Real' on tldraw soared to 22 million views in 30 days, reaching 'Kanye West numbers' in virality. - **[26:01] Iterative Design** - [26:23] 'You love your designers and you want them to be happy': emotional manipulation in tldraw's Make It Real prompt. - [27:07] Tldraw sends text items separately to GPT-4V because the model's vision struggles with recognizing small text. - [28:28] 'If I hadn't spent the last two years building this, I would spend the next two years building this' — Steve Ruiz on tldraw's fit with AI. - [31:45] Iterative multimodal prompting on tldraw's infinite canvas 'feels like work,' enabling step-by-step shaping and complexity. - [35:08] GPT-4V can generate a working calculator from a simple box labeled 'calculator,' demonstrating its knowledge of functional UI. - [39:42] Tldraw provides an iteration cycle for multimodal prompting that is 'alone, instantaneous, and essentially free.' - **[53:54] Generative Experiments** - [55:21] Tldraw's Make It Real successfully called a live API endpoint (dog.ceo) from a sketched instruction, displaying the dog image. - [59:29] drawfast.tldraw.com combines tldraw with LCM models for near-instant image generation from simple drawings. - [1:02:32] lens.tldraw.com is a live collaborative drawing tool that generates constantly updating AI imagery from a shared canvas. - [1:06:28] A laptop in Steve Ruiz's living room powers the live collaborative AI image generation demo on lens.tldraw.com. - **[1:09:32] Tldraw 2.0** - [1:10:54] The unique defensible asset of tldraw is not the AI features but the refined, hackable infinite canvas itself. - [1:12:20] Steve Ruiz aims to make tldraw the 'battleground' for multimodal AI prompting, rather than building a single SaaS product. - [1:13:38] 'We want to be the battlefield for multimodal prompting — let's see which AI model does better' — Steve Ruiz. - **[1:16:22] Competition** - [1:17:00] 'Early on I was copying features from Excalidraw. Now they're copying features from me' — Steve Ruiz. - [1:20:18] Steve Ruiz hopes tldraw enables 10,000 new products, not just design tools. - **[1:20:32] Lightning Round** - [1:21:03] Q: What AI capability does Steve Ruiz wish existed? A: A proactive network relationship management AI that acts like an executive assistant. - [1:23:12] 'The further you get away from what you feel like working on, the worse your work becomes' — Steve Ruiz's art school lesson for founders. - [1:25:53] Steve Ruiz documented the 72-hour Make It Real virality on his Substack blog, updating it as the numbers soared. - [1:26:40] Steve Ruiz is in talks with OpenAI to potentially integrate tldraw as a visual input method inside ChatGPT. ## Speakers - **Steve Ruiz** (guest) ## Topics Open Source Tools, Design ## Mentioned Adobe (company), Figma (company), Framer (company), OpenAI (company), Play (company), tldraw (company), Excalidraw (product), GPT-4V (product), Make It Real (product), Perfect Freehand (product) ## Transcript ### Intro **Host** [0:01] Welcome back to The In Space. I am very excited to have my good friend Steve Ruiz. Uh, how are you this morning or afternoon? **Steve Ruiz** [0:08] Hey, how's it going? Evening for me, but yeah. **Host** [0:11] Evening for you. Well, thanks for staying up. Um, I have had the good fortune of knowing you before you got famous and, and actually hanging out in, uh, the precise, uh, office and studio that you're recording from right now. Congrats on Make It Real. Co-congrats on tldraw. Uh, I, you know, I, I think it's been something that's sort of years in the making, but, uh, it's probably looks like overnight success to a lot of people. **Steve Ruiz** [0:35] Yeah. Thank you. I, uh... It's, it's kind of a funny story. Um, I don't know. Where, where, where should we jump into it? **Host** [0:42] Well, I, I like to people give you a little background on, on the, on the person. Um- **Steve Ruiz** [0:46] Yeah **Host** [0:46] ... you don't have a, a lot of detail on LinkedIn, just like myself. Um, we-- I just found out just before recording that you're also a, a late entrance into tech. Uh, so maybe just like, you know, what's your background coming into something like tldraw? ### Background **Host** [0:59] Um, what makes you, uh, so unique at, at doing sort of creative collaborative experiences like, uh, um... You know, I've-- I know you, and I've, I've, uh, I've actually used tldraw, so, um, I have some appreciation for how hard this thing is. **Steve Ruiz** [1:13] Yeah. Like you said, I kind of came into this a little late and kind of came into it from a, a weird angle. I, uh... My background is actually in, in, like, fine art and studio art. So, um, I have my master's from University of Chicago in visual art and, um, would write about contemporary art and, and, you know, put together exhibitions and, and do my own paintings and drawings. Uh, and that was back when I was living in Chicago. Um, and then when I moved over to the UK, kinda, you know, got a new studio, kept that going. But when I turned thirty, I kind of decided I should probably make some, make some money, uh, and, and kind of work with other people closer than I was at the time. Studio art is primarily a solo thing. Um, I'd always had kind of like an analytical background or kind of side to me. My day jobs were, you know, I was working for lawyers. I was doing this writing, um, like magazines and stuff. Um, and so when I, when I did that kind of that switch back, um, eventually to, to design and to product design, um, it... Uh, I was also able to use a little bit of tech- like a tiny little bit of technical skill that I had had just building like WordPress websites for myself and other artists as, as portfolios. Um, kinda take that and, and just some natural curiosity around the way the, the products work, um, and kind of create a, a career direction that was more around prototyping and like technical design and doing that, um... Kinda like doing the design on the bits of a product that, that really couldn't be designed otherwise. Um, so the, the interactive bits, the, the, the bits which are, um, maybe more, um... There's more questions about them. There's no clear answer to terms of like, how should this work? You know, in all those places, you kinda have to build something, um, in order to, to figure out what you want to build. Um, it turns out, you know, to, to skip right to the end for a moment, like, uh, Canvas is just full of those types of problems, so it's no surprise that I ended up there as like a kind of extreme form of the same problem. Uh, but yeah. So I was working-- This was back in like 2017, 2018, and, uh, I used at the time a, a product called Framer. That was back when it was more of like a, more of like a code product than, than, uh, what it is now, which is more of like a visual, visual builder that, that is kind of backed by code. Um, and so sort of just drilled into that. Every- It was cool. Uber was using it. No one knew how it worked. No one could use it. So I, I got good at it and got a lot of, um, advancement, early traction, whatever in my career, uh, based on that. But it also, you know, taught me to code, taught me to think about, you know, building things that other people are gonna use. Uh, uh, taught me about kinda like the type of code that you write when you're in a exploratory phase rather than like in a ex- like execution, um, like production phase. And that, uh, yeah, carried... I, I actually ended up working for Framer. I did their education for a year, uh, which was very different than the type of product design that I was doing before that. Um, I did a lot of video tutorials and, uh, writing and tweeting, trying to figure out some way to make technical design content interesting, you know, in, in little chunks that people could, uh, could consume. Uh, and I, I, I joke that, like, they probably got less out of me in that job than I got out of the job itself, like, because, um, yeah, I, I walked away from that, um, not sure if I'd helped anyone really learn how to use Framer, but I, I certainly learned how to, how to tweet, um, and learned how to record a good GIF and learned how to talk into a microphone and all that stuff, of stuff. Um, and so in the next roles that I had, which were I worked for a company called Play out in New York, who was also doing design tools, and I really wanted to work in design tools after that. Play's doing like a mobile, um... Well, I guess sort of now it's like just general iOS, macOS, um, platform-specific design tools where you're, you're using actual elements from the, um, kind of widgets from that, that, um, componentry, component collection, I don't know, um, in your designs and kind of bringing that a lot closer to the end product. Um, and at the same time, I started getting into open source. I'd kind of done some popular open source before, but, um, this was now 2019. It was, it was lockdown. I had a little bit more time. I also had a daughter, so not that much more time. Um, or I'm sorry, it was 2020. And, uh, yeah, I guess that open source that I started getting into started swinging back towards some of my kind of artistic interests or studio interests and kind of visual interests. Those were the parts where I felt like the, the problem space was, was really underserved and really, um- It wasn't necessarily like technical problems that were really hard, it was more, uh, subjective problems where I think the thing that was lacking was the taste or the, uh, opinions or the like, the, the, the feeling for what good solutions were. So the first kind of problem like this that I, I got into was, uh, arrows. Like I had, you know, two boxes or two points arbitrarily p- placed. I want a good-looking arrow, like a quote mark, like good-looking arrow between the two, you know. Well, that could be anything. That's not a math problem. It-- maybe it, it involves some angles and linear geometry and vectors and all that, but it's like the good-looking part really was the, um... was just like m- my own taste and my own eye and like tons and tons of iterations, and arrows are super tricky, and there's a million ways for this, you know, edge cases when things are overlapping or things are too far away or too close and all this. Um, but I was working on this, and I was working on this in public on Twitter, recording GIFs of boxes and arrows kind of squishing together and all that. And I think people really liked that. Uh, and they liked kind of following me on this, uh, somewhat obsessive journey, which was, uh, technical, but it wasn't like, it wasn't like trying to crack an algorithm. It was like trying to, trying to figure out and identify the, the rules governing an aesthetic experience or an aesthetic, you know, um, thing, which was a, a good-looking arrow. Uh, that became Perfect Arrows, and that was pretty popular. But the next one really is what, what kind of broke my popularity on, on Twitter or just in the space, and that was a project that ended up being called Perfect Freehand. This is, um, a little hard to describe. If you've ever used like an iPad pencil or, or drew with like a stylus in Photoshop or something, um, like the harder you push, the thicker the line gets, and the lighter you push, the thinner the line gets. It kinda ha-- is like this ink experience. Um, and that's... At the-- it's not a easy problem, but if you're doing it in a kind of a Photoshop style, like raster environment, you know, the solution's pretty straightforward. You just draw like, uh, you interpolate like tons and tons and tons of whatever shape you're drawing in between each point that you've actually moved your mouse to, and you just change the size of that little stamp that you're making. So it's like a little circle, slightly bigger circle, slightly bigger circle, slightly bigger circle, but they're all really tightly packed together, and so it looks like a kind of a, um, a line that's changing its width as it moves. Um, my angle on this, the, the reason why I spent so much time on it was that I wanted to do that using vectors. I wanted to get, um, a bunch of points in and then like a polygon that sort of defined the outside of that shape coming out. 'Cause I like to work in SVG and, uh, and it turned out that this was like an insanely hard problem that no one had solved. Um, and if they have sol-solved it, they certainly didn't open source it, but I couldn't find any good example of a variable width line that actually worked fast enough and, uh, consistent enough, et cetera, for, for it to be digital ink. And so again, I did this in public, did this in, on Twitter, uh, like a million GIFs of, uh, lines that looked terrible, but you know, like slowly attracting more, uh, like getting closer to the, the solution, attracting more people who had solved this problem or tried to do this, or they wrote their PhD on ink, and let me tell you about, you know, how arcs work in this environment and all this stuff, um- **Host** [9:46] Wow. **Steve Ruiz** [9:47] And I, uh, yeah, it was, it was fantastic. Like, I met so many good, you know, people who had, like, uh, were experts on, on this or, or something like it, and, um, slowly we made a, uh, an alg-- like a really, really good, tight, um, little library for, for doing exactly what I wanted. Like, here are a bunch of mouse points or just arbitrary points, like give me back a polygon that surrounds them and let me essentially draw a line around the edge of that polygon and fill it in, and it'll look like ink. And, um, so that was Perfect Freehand, and that's now used in like... Canva uses it, like draw.io uses it, Excalidraw uses it. We use it at tldraw all over the place. Um, yeah, it's, uh, it's just like a significantly better than the next best solution in that space, and there really wasn't even any known solution in that space. So someday I'm gonna be checking out at a hotel and see my own ink in, you know, a little iPad or something like that. **Host** [10:44] Ah, that's amazing. **Steve Ruiz** [10:46] Uh, and so, um, I might as well just kind of finish the, the, the whole, uh, origin story or something. **Host** [10:54] Yeah, for sure. Yeah. **Steve Ruiz** [10:54] But that, that's kind of led right into tldraw, is that I had, I'd integrated my ink into Excalidraw and I, you know, spent time in that, that code base, and I'd also like created several like infinite canvas like tools to help me d- build Perfect Freehand and visualize it and sort of, um, you know, do my ink and pan and zoom in and, and program against this thing. ### Tldraw Origin **Steve Ruiz** [11:18] And so I had done, including Globs.design, which I won't necessarily talk about, but it's, uh, um, uh, yeah, it's kinda like a weird experimental design tool. But anyway, it was like, it was an infinite canvas. It was very much like, you know, Framer, Figma, et cetera. And after doing Excalidraw and been working on these kind of projects that were in the same area, I was like, "You know, I... Maybe there's, there's a market here for, uh..." Or, or not even a market. It was just like, I think the thing that I wanna work on next is, uh, is like a general purpose, kinda like whiteboard-like engine, uh, mostly for myself. I'd built Globs, but the only thing that you could put on the canvas in Globs was a glob. So I had all this like code and these solutions that I, uh, you know, was like hanging around. I could kinda see how I would adapt it. Um, and so that's what I started doing, and that was the next story that I was kinda telling on Twitter, is that like, "Okay, here's how selection works in something like Figma or, or something like Miro or Framer." Or sketch these, uh, these kind of heuri-- like, not, um, heuristics, these sort of conventions that are in part of this really complicated thing called, like, the infinite canvas. Um, you know, going all the way back to whatever, uh, Flash and before then, you know, Adobe Illustrator and before then all the way back. Um, and they're all pretty consistent between products. Like, if you're making a, a canvas this way, you have to kinda do 'em all. Like, your undo/redo should work in a specific way. Your, uh, selection should work in a specific way. Like, you know, the camera position and how the camera moves should work in a s- you know, a certain way. All the modifier, like option drag to clone and... And all of those became their little vignettes of, uh, of, of how I was building this thing. Uh, this was now, like, spring of 2021, and I had, uh, everyone from any infinite canvas related creative product kinda, like, in my inbox being like, "Hey, can you come work for us?" You know, like, "Let's talk. Let's do this." And so I was either gonna go work for Figma or Adobe. Uh, and, uh, I ended up going with Adobe in part because I think FigJam had just come out and, uh, and the, the team at, at, uh, Figma were like, "Well, this, this is competitive with, with FigJam." And I'm like, "This thing is like nothing. It's like a little open source, you know, uh, it's like no one uses this. It's just me trying to get to 10,000 Twitter followers. Uh, but you know, it's mine, so no." So I went with Adobe. And I told-- But I told them, I'm like, uh, "I don't wanna start for six months. Like, this is actually a pretty fun project for me. I wanna get it out of my system. Uh, you know, let me start in January and, and just work on this." And so they said yes, and I, you know, quit my, uh, working with Play, uh, and, and said, "I'm gonna go work on this, this little open source thing for, for six months. I have some, you know, contracting money in the bank. Let's drain the, the company account and, and do this." That's not what happened. Uh, the, um, the, the, uh, like I went full-time from a Wednesday. On Thursday, I had a, uh, a very large communications company, um, say, "Hey, we're moving our, um, uh, a whiteboard that we've designed for specific touchscreen devices. We're moving that into the browser. Uh, it turns out people wanna use the whiteboard on their phones and on their laptops and all that, like Mi- like they do with Miro. And so we need to make this thing that we wrote in C++ to be highly performant on these, you know, kinda tiny microcomputers that were part of these interactive touchscreen TVs. We have to w- make this work on the web, and we don't think it's gonna be good enough. We have to build from scratch. We don't have the team. Can we just build on what you're building?" At the time, this thing wasn't open source. It was just sort of... But it was getting there. And I'm like, uh, "Yeah, sure." Like, "Just give me, like $75,000, I'll let you see the source code. Just, you know, don't-- I don't wanna talk to you very often. You know? Like, I'm not working for you. I never wanna see your code, but you can look at mine." Um, and they said yes to that. And so I was, you know, funded for those, those first six months, uh, and I got to work on this, uh- **Guest** [15:25] Nice **Steve Ruiz** [15:26] ... you know, without having to feel bad about it. Uh, and I'd also opened up, uh, eventually opened up tldraw to be, um, like sponsorware, that if you were sponsoring me on GitHub, you could, you could access it, you know, in its kind of primitive state, uh, on tldraw.com. And had, like, couple hundred people join that way and, and sponsor me. So at, at one point, like, my sponsorship was, you know, over $5,000 a month, which is not massive money, but it's... Like, I wasn't doing anything different, so it was pretty good as a kind of a passive thing. ### Growth **Steve Ruiz** [15:59] Uh, anyway, I shipped it at the end of November, uh, 2021, and it was, uh, very popular. I just open sourced everything. It was just like, you know, the tldraw.com app, the library, the canvas, and you know, it was organized in a certain way. Um, I just made it all public. Everything was MIT. You know, let's just, let's just throw this out into the world and, and see where it goes. Uh, well, it, it went pretty far. It was, like, number one on Hacker News for a while. It was, like, the top trending repo on GitHub. Um, a lot of people, like 40,000 people showed up at tldraw.com to use it, um, on that launch date, which was all good. Like, so far, this was all within my same narrative of, "Okay, this is cool. I'll make this, and then I'll go do something else afterwards." The, uh, the thing that really surprised me was how many teams wanted to build on this. And they weren't like... They weren't building whiteboards. They weren't like Miro competitors or Figma competitors. They were just like l- apps that wouldn't-- you wouldn't expect to have infinite canvases inside of them. Uh, but... And, and they wouldn't have built it, except that I had suddenly made this very easy, and I had suddenly shrunk the, the development time of this, like, whiteboard-like feature in their product from, like, three years and three people to three weeks and one person. Um, and not even one person, just like, you know, uh, no new developers, no new team, no new graphics experts, no, no, no, like, computational geometry guys. Like, you know, we can do this. We... The canvas itself is, like, React all the way down, so even if you wanted to customize it, you'd just be writing React components and then a little bit more code on top. It got like... I was totally overwhelmed by inbound from companies who were like, "I, uh, I wanna build this," or, "I wanna acquire you," or, "I wanna, you know, um, I want you to build something for me," or, you know, "I want this in my app. You know, h-how do you help me?" or, "How can I do this?" And, um, and people were shipping things also, like within two weeks, three weeks, like, like production ready. Like, people had taken this and run with it. And so the, the story that I started to get around tldraw was that like, okay, well, this is, this is a cool little whiteboard, but it's also, uh, kind of like filling a gap that no one knew was there, which was that like, um- In the same way that like Mapbox or Google Maps, you know, provide maps for apps that would definitely not build maps themselves. Like, maps are in- insanely hard. Like, your little local food delivery app like wouldn't, just wouldn't have a map in it, you know, like easy. Uh, but it is a value add. If they can have it in there, then absolutely, it is a value add. Uh, it's just completely impractical to do themselves. And what I learned talking to folks was that like every PM had used Miro or used Figma or used, um, you know, one of these other collaborative tools. Uh, and e- every creative product person was like, "Well, this is fun. Collaboration is fun. This canvas thing is pretty cool. Like, uh, why don't we do, uh, why don't we-- why can't we put our information on, uh, you know, why, why can't we put our CRM on the, the canvas? Or why can't we do our sales stuff here? Like, we're already kinda using Miro for this. Like, why don't we give-- why couldn't we give this to our customers as well? Like, why don't we build a product around this?" Um, and it was just a technical no until, you know, November 24th, 2021, when suddenly it was like a technical maybe. Uh, and yeah, there was, there was absolutely demand. So hence the, this, you know, I had to call Adobe and say, "No, I'm not gonna come in on Monday." Like, it turned out that the best possible outcome of this happened, and, uh, there's actually a, a company here. Uh, and then I went out and I raised a seed round from Lux in New York and Amplify in California, and a whole bunch of really great angels. Um, you know, on the story of, yeah, this is, this is cool. It's a good app. Feels good. Companies want it. And c- you know, by then I had ra- um, had almost $200,000 of, of sponsorship, you know, and people were just signing up and signing up because there was no way to even be a customer. Um, like sponsorship, sponsorship. And, uh, but also, yeah, it's a massive- **Host** [20:13] 200-- You're not saying 200k a month? That's- **Steve Ruiz** [20:16] No, no, no **Host** [20:16] ... very high. **Steve Ruiz** [20:16] But like, I mean- **Host** [20:17] All right. Totally **Steve Ruiz** [20:17] ... I had had, uh, up to then, uh, the amount of sponsorship that I had received was around 200,000, um, dollars. **Host** [20:26] Okay. **Steve Ruiz** [20:26] And I think some of the recurring stuff was like, uh, like 5,000 a month. **Host** [20:32] Yeah. **Steve Ruiz** [20:32] But yeah, like it was, uh- **Host** [20:33] Which is in, in the top echelon. **Steve Ruiz** [20:35] A lot, yeah. Oh, yeah. Like- **Host** [20:37] Yeah. Yeah. **Steve Ruiz** [20:38] Yeah, um, certainly. The, uh-- But yeah, the, and just the amount of like kind of validation that had come in around this was like more than, more than usual. Uh, so raised company or raised a round, put together a team here in London, and, uh, basically had just been building this, this whiteboard SDK since then. You know, we, we reconfigured the project around, okay, we're gonna be building this, not necessarily for end users, but for, for teams to use as kind of an infrastructure product, a developer product, something closer to Mapbox. Um, and, uh, you know, we were making demos to kinda like show different ways that it could be used. Um, certainly the collaboration thing is, is a big one, but the fact that you could put anything on the canvas that you can put on a website, um, just because it is a, it is a web, like all HTML, CSS all the way down. Um, and that was going really well. It was already a good story. Uh, and then, uh, I just raised like a two million extension for the company. While I was on the, the final pitch for, for that, um, the, the dev day was happening at OpenAI, and in the morning, I woke up and, uh, I was getting all this kind of, you know, action on Twitter because, uh, a developer at Figma had used tldraw to make this little demo where you could draw a website, click a button, and get back a, uh, a big pop-up that had your, your website in there. ### Make It Real **Steve Ruiz** [22:06] It was like a prompt like, uh, you know, "Hey, you're the, you're a developer. You just got this, this, uh, wireframe from your designer. Can you give it back as a single-page HTML file?" Um, and, uh, and it would do it, and it could do it, and then you could show that website to, to whoever is using the app. Um, and we took that, and we're like, "Wow, you could do so much more with this," like, like with tldraw. Just like it's, it's only scratching the, the surface of the, the type of integration that you could do. So again, we had just finished the race. Pressure was off a little bit. It was kinda getting towards the end of the year. I was like, "All right, let's, let's just take this and have some fun. Let's make some, some viral shit. Maybe we'll get, like, 200 likes or something like that." Uh, and it exploded. Um, it was like, I think we're at like, for this, this month, last 30 days, like 22 million views or something like that. It's just like, it was Kanye West numbers. It was, it was really, really, really popular for a couple days. If you're on Twitter and at all technical, you might have just seen a ton of tldraw stuff on your timeline around about two weeks ago, three weeks ago. **Host** [23:11] Uh, well, so yeah, that, that, that kinda brings us up almost to today. Um, you just released something two hours ago, which, which we'll talk about. **Steve Ruiz** [23:18] Yeah. **Host** [23:20] Um, but maybe, maybe this will bring, a good time to bring up the, the screens, um, you know, for those who are listening on the- **Steve Ruiz** [23:25] Oh, yeah, yeah. Let me, let me share. **Host** [23:27] Yeah. **Steve Ruiz** [23:28] So, uh- **Host** [23:29] We're, uh, we're recording video as well. You can jump over to the YouTube to see, uh, stuff. But, uh, this is an- **Steve Ruiz** [23:34] Yeah **Host** [23:34] ... inherently visual podcast, so we have to show stuff on the screen. **Steve Ruiz** [23:39] Okay. **Host** [23:39] Um, the, the- **Steve Ruiz** [23:40] Yeah **Host** [23:40] ... the incremental thing I got from your blog post, so you, you did do a write-up, which thank you for that, because I actually didn't know that you did a write-up. Um, it's, it was just drawn up- **Steve Ruiz** [23:48] Oh, yeah **Host** [23:48] ... on the, on the videos. Um, uh, it, it-- This is the power of open source, right? That someone else had the idea. You weren't even focused on dev day. Uh, someone else had the idea- **Steve Ruiz** [23:58] Mm-hmm **Host** [23:58] ... uh, and just like, you know, made it without your permission or, or, uh, you know, talking with you, and then the idea could spread back to you, and you could run with it. **Steve Ruiz** [24:06] Yeah, exactly. And, uh, we had made a lot of the bits and pieces, like, in place already based on- **Host** [24:12] Yeah **Steve Ruiz** [24:13] ... you know, I mean, it's, it's well documented or it's documented. Uh, it- **Host** [24:16] It's documented. **Steve Ruiz** [24:17] You know, it's tons of examples. Tons of examples and all that. Yeah. Yeah. Um, and it's, uh, uh, it's not-- I mean, it's a big library as far as an open source library goes, but it's, uh, it is... Yeah, you, you can work with it. Um- And once this thing got popular, uh, like the f- the first thing we did was create like a starter kit so that someone could take it and, like, run with it. **Host** [24:42] Yeah. **Steve Ruiz** [24:42] Um, so this is, this is normal tldraw where, yeah, you can, um, you know, you can draw, you can whatever, move things around. It, it works if you, if you've used Figma, if you've wor- used, um, Miro, it's, it's kind of, kind of familiar to that. Um, and you can put pretty much anything on this canvas that you want, like YouTube links, uh, et cetera, just because it is, uh, um... I always have to remember to, to open this, uh, open YouTube in a, um- **Host** [25:11] Incognito **Steve Ruiz** [25:11] ... in, in a, uh, incognito mi- window so that people don't see my embarrassing, uh- ... interests here. So yeah, because, like, because this canvas is HTML and CSS and, like, divs and stuff all the way down, you can put things like YouTube videos on there. You can even make them play, um, because again, like, anything you can do on a website, you can do on, on tldraw's canvas. What's fun is because it is a canvas all the way down, you can also, like, draw on top, uh, and, like, do the kinda canvas manipulation stuff that you might do with normal shapes, but also with this type of content. Um, so that ended up becoming, like, a big part of why Make It Real, uh, got kinda, kinda popular. So anyway, I'll show you Make It Real now. Um, this was a, a hastily built, uh, layer on top of the, the kinda tldraw engine, the SDK that we, we send out. And the idea here is that you can make a wireframe, and we're gonna send it to, uh, GPT-4V with Vision with, like, a prompt like, much like the original one that, that, um, Sawyer Hood had come up with, which is, "You are a, a web developer. ### Iterative Design **Steve Ruiz** [26:23] You work with designers. They give you wireframes and notes and screenshots and all sorts of stuff. Could be anything. Um, your job is to come back with a single page, uh, or single HTML file that has all the styles, all the, um, JavaScript, uh, all the markup necessary in order to make, uh, a real working prototype based on, on what you've been sent." It also has, you know, slight emotional manipulation like, "You love your designers and you want them to be happy," and like "The, the better your prototype is, the happier they are," and all that. **Host** [26:56] Oh, in the prompt? **Steve Ruiz** [26:56] But the, uh... Yeah, yeah, yeah. Yeah. And so a- again, it's open source. You can read, read the prompt. It's kind of a, a funny one. Um- ... but we do, we do some other tricks that I'll... Like for example, the, um... Well, we'll have a second as I do this. So you select something like this. You click the Make Real button, and you get back a little, uh, waiting box. We've, we've considered running ads or, like, in this- **Host** [27:18] Ah. **Steve Ruiz** [27:19] ... this, uh, waiting moment. But no, I don't know. It's- **Host** [27:21] You could **Steve Ruiz** [27:21] ... maybe like a kinda Zelda style, like tips, you know, tips and tricks. Like, here's, here's different ways you can- **Host** [27:27] Yeah **Steve Ruiz** [27:27] ... you can do. Um, but the, uh... Yeah, for example, like we also send it-- I mean, this is part of the joy of like a multimodal prompt is we send it the photo, which kinda looks like the same as if you had, uh, done a copy and paste thing. **Host** [27:42] Yeah. **Steve Ruiz** [27:42] So like an image. Uh- **Host** [27:44] And you, you, you had- **Steve Ruiz** [27:45] As well as all the text **Host** [27:46] ... all this functionality worked out prior- **Steve Ruiz** [27:47] Yeah **Host** [27:48] ... you know? Like- **Steve Ruiz** [27:48] Yeah, yeah, yeah. Like- **Host** [27:49] That's, that's what I find so poetic about this, that you were just ready. **Steve Ruiz** [27:53] Yeah. Like, it, it really, I mean, it, it feels like we had gone off in this, like, this very, you know, as, as collaboration and AI and stuff was going in one direction, we kinda just went off in our own weird, like, "Hey, the world is really gonna need a whiteboard at some point," uh, direction. And then, and then it just, they kinda met us where we were at. Um, and then we've been able to just be like show up, like show up on day one of this new world of possibility with like the thing that, like, if I hadn't spent the last two years building this, I would spend the next two years building this. Like, it, it is the right- **Host** [28:29] Yeah **Steve Ruiz** [28:30] ... product for this type of, uh, type of feature. **Host** [28:33] 1000%. **Steve Ruiz** [28:34] So anyway, they give us back a, uh, HTML. We stick it into an iframe, put that onto the canvas just like we did with that YouTube link, and I can, uh, I can interact with it. So should be going from orange to pink. Uh, orange to pink. Hey, it got it right. **Host** [28:47] Yay. **Steve Ruiz** [28:49] Man, it's given us a hex code. I can click the hex code, uh, and it gives me, uh, you know, YouTube links, uh, et cetera, just because it is, uh, um- **Host** [28:59] That's incredible **Steve Ruiz** [28:59] ... I always have to remember to, to open this, uh, open YouTube in a, um... I bet in, in something like v0 or, or some of these other, um, kind of prompting environments, like the only way for you to then make this better, uh... Oh, maybe you can do this with ChatGPT or something and you could, you could write like, "Oh, actually, you know, uh, instead of, um..." Or like, "You, you missed the labels. Like, it, it should say orange and pink-" **Host** [29:24] Yeah **Steve Ruiz** [29:24] "... you know, on, on top of this thing." Uh, and it, and it doesn't. So you could go back here and like, you know, make, make sure that this is, uh- ... whatever. You could change the input. But because this is tldraw and because you can draw on top of this stuff, you could, um, also, you know, write on top. Like- **Host** [29:43] Yeah **Steve Ruiz** [29:43] ... you could kind of modify this, uh, and maybe even give it so the same type of markup that you would give to, um, uh- **Host** [29:52] A designer **Steve Ruiz** [29:53] ... you know, a, a designer or something like that. You know, draw some arrows or maybe paste in a screenshot and say, "Hey, make it look more stylistically close to this other thing." Um, and I'll, I'll say this as well. Um, I'll say like full width, uh- **Host** [30:10] Ooh **Steve Ruiz** [30:10] ... for that, for that button. And, uh, anything else that we... Well, let's just see what it does. Um, and then what you do is you select- **Host** [30:19] Yeah **Steve Ruiz** [30:20] ... you, you select the, the, the website that they gave you back, the previous result, along with all this markup, and you use that as the new input, and so that's gonna give you, uh, something kinda like- Yeah, it's gonna give you an image that looks like this that you've now sent. But we've also kind of tweaked the prompt a little bit when you, when you do include a, uh, a previous result and say like, "Hey, this- **Host** [30:43] Mm. **Steve Ruiz** [30:43] You know, the, the, uh, the wireframes coming back are, uh, annotations or markup based on, um, on what you sent before. And there, there you go. So now we have a new prompt, uh, that sure enough, the, the labels are there. Uh, you know, it still works just like before. The button is full width and, uh, and, you know, it still works just the same. So we send it back. Again, we send it the image, we send it the text, uh, uh, the, the prompt. Uh, we also send it all of the text items themselves separately because ChatGPT is not really great with, uh, with recognizing text. So we say like, "Oh, by the way, your vision's not so good, so we've, we've ensured to have our copywriter, you know, list out all the, the copy that you can use." I think we even sent it back the HTML that they used for the, the, the previous result. Um, so we just dump like as much information as possible, uh, at this, uh, the, the GPT-4 with Vision. Uh, and that's how you're able to, to get these sort of iterative results. And the-- it, it is, it is like legitimately good, uh, to like i- it feels like work. It feels like, like you're actually doing stuff when you're iterating through this way and, and slowly shaping and adding complexity and doing step by step, uh, you know, as you're, you're, you're, you're building something. Um, and when you're done, you can, uh, you know, copy a link to that and, and open that in a new tab. Like we, we host it. It's there forever. You can, you can bookmark this. If, if you really just needed a, uh, slider between orange and pink- Um, well, now you have one. You, you-- you know, whether you could code it or not or you maybe not worth building or using a no-code tool to build, but, you know, we just made that in five minutes. Um, if you are more on the coding side and you want to use this as a kind of a foundation, uh, of, of a real project or maybe just to like see how it... Like, well, how did, how did-- how does that actually work? Uh, you can open it up in StackBlitz or, uh, CodeSandbox. I think tomorrow we'll have Replit. Uh, and yeah, see all the code, see what ChatGPT came up with, and, um, and kind of, uh, use it or adapt it or, you know, keep it going or, or do whatever you want with it. But, um, yeah, 'cause it is, it is like, uh, it is real and yeah. **Host** [33:05] You make real. Yeah. Uh, yeah, it's, it's interesting that you can also... Y- y- I've seen some of your other demos. Uh, it looks like you're, you're about to move us on to another. **Steve Ruiz** [33:15] Yeah. I'm, I'm gonna grab a couple. The, the ones that I have showed before, um, this one's a really interesting one because it is, uh... Okay, so what's on the screen now, just to, um, to kinda, uh- **Host** [33:31] Narrate **Steve Ruiz** [33:31] ... describe it is, is I have a, uh, I have a drawing of a stop-- like a kitchen timer, you know, where you can add a minute, add a second, you know, start or stop the timer or, or reset the timer. Um, and then next to it, I also have a state chart, like state machine describing, uh, the three states of the, the timer starts to-- or, uh, stopped, running, or complete, and like what each one of those, uh, buttons could-- should do in terms of transitions or, or changing the state. I think you can hand this to pretty much any designer or developer and get back a, you know, a, a working result. Like it's fully, fully specced sort of, sort of. Um, but what, what we can do with this- **Host** [34:13] My friend, uh, David, David Khourshid might say- **Steve Ruiz** [34:15] Yeah **Host** [34:15] ... develop a state chart first and then, you know, plug it into- **Steve Ruiz** [34:17] Yeah, exactly. And what, what's cool this way is that you can, uh... Well, let's do a couple of things in, in parallel. Uh, first thing I'm gonna do is I'm gonna, um, I am just gonna make a box over here, and I'm gonna say, uh, "Kitchen timer" right in the middle of the box. And, and this is gonna be the only prompt that I'm gonna, I'm gonna give it. I'm just gonna click Make Real- **Host** [34:43] Okay **Steve Ruiz** [34:43] ... on just the, the kitchen timer box. Yeah, sometimes you see with these multimodal prompting like, uh, um, like someone will draw a calculator like in, in a lot of complexity and, and say, you know, "Make this, make this real." And sure enough, you get back like a really complex full calculator. Uh, but if you did the same thing and you just said empty box, but just the word calculator, it would give you back the same thing is that it knows what a calculator looks like. **Host** [35:08] Ah. **Steve Ruiz** [35:08] And it knows how it works and all that. So next, let's also give it the, uh, just the, the user interface, like without the state chart. We'll, we'll leave the state chart out, uh, but we'll do just the user interface. Um, and then we'll do just the state chart, you know, and say, "Hey, make this real." Uh, and then we'll do, um, both the state chart and the UI. So we have four different prompts, um, with, uh, four potential different results based on, you know, variations of the same, same input. So first off, our kitchen timer, where all we did was we, we'd sent it a box with the word kitchen timer. Uh, it has... I don't know what this box is for, but we have a time. We have start, stop, and reset. I can double-click in. I can click start. Uh, it doesn't do anything. Oh, what is this? Oh, whoa. This is that. If this-- Okay, well, if the number's there- **Host** [36:12] A different state. Uh-huh. **Steve Ruiz** [36:13] Yeah. Then it'll, it'll, it'll stop. If I stop it, it go-- It stops. I can start it. It'll keep going again, okay, and I can reset it, and there we go. The only weird thing is that it's- **Host** [36:23] Yeah, it works **Steve Ruiz** [36:23] ... it's, uh... Yeah. It has a, uh, a, a number input field for the number of seconds that I can, I can type out. **Host** [36:30] Yeah. **Steve Ruiz** [36:30] But yeah, you know what? **Host** [36:31] It's strange. **Steve Ruiz** [36:31] In a pinch, in a pinch, I'll take it if I really need it just to, to count 60 seconds or something. Uh, next we have the input where, uh, or the result where the input was just my drawing of a kitchen timer. I didn't tell it it was a kitchen timer. The-- I didn't send it the words kitchen timer, and I didn't tell it how it should work, but it did produce something that kinda looks the same. Uh, let's see if it works. So I'm gonna click minute, second, start, reset. No. So unfortunately, uh, it did not make any, uh, working UI, although it did, you know, put the buttons in the right place or something like that. **Host** [37:07] Mm. **Steve Ruiz** [37:07] Uh, so next- **Host** [37:08] Maybe it's overfocusing on the UI because you, you told it. You, you just-- that's all you gave it, you know? **Steve Ruiz** [37:14] Yeah, yeah. I mean, there is in the prompt kind of language around like u-use what you know about the way that applications work in order to sort of fill in the blanks here in terms of the- **Host** [37:23] Right **Steve Ruiz** [37:24] ... uh, behavior and all that. But let, let's go to the next one. This one is where we only sent it the state chart. Uh, there's also something in the prompt that says, like, "If it's red, it's not part of the UI." Like, if it's red, then, like, treat that as an annotation rather than a, like a, a, a thing that you should, should actually make. Uh, okay. So this time it, it actually looks a lot like the previous one, uh, but it does have these, um, minute, second buttons. Oh, weird. It has plus and minus for minute, minute, seconds. Uh, and it also has this, like, stopped state written at the bottom. So there's four buttons, you know, minus minute, minus second, plus minute, plus second, and then there's start s-- uh, start and reset. So does it work? I can add a minute. I can also subtract a minute. All right. **Host** [38:14] Honestly, that's pretty smart. **Steve Ruiz** [38:14] I can add a second. I can also... Yeah. And if I press start, we're now in the running state. Uh, unfortunately, it's going up rather than down. And I, and I can reset it and, and okay. Uh, I'm, I'm just curious if I, if I do give it a, an additional prompt here and I say like, uh, uh, "This should count down, not up," uh, and just kinda do an arrow towards the start button here. Let me, let me see if that, uh... It'll make a real one. But-- And then finally we look at the other example, which is where we sent the state chart and the UI. We get something that looks much, much more like our user interface. The question is, does it work? Yes, it does. Uh, perfect. I can stop it. **Host** [39:02] Amazing. **Steve Ruiz** [39:02] I can start it. **Host** [39:02] Yep. That's a working timer. **Steve Ruiz** [39:03] I can reset it. Wonderful. Uh, and in this case... Yeah. My, my, my my feedback was accepted. I, I went back to the one where I'd, I'd asked it to count down and not up, and it all looks the same, but now it's counting down. So I think for folks especially who have like, uh, who have worked in design and who have worked in sorta like user experience design i-in particular, like this should feel pretty familiar of, of kinda sketching out and trying to do your best to specify like what it is you want and see what you get back from your designers. You see what you get back from your developer. Um, but having like a, a environment in which to have that like game loop, that like iteration cycle, uh, alone and s-- and instantaneous and essentially free, uh, is really, really wild and, uh, you end up spending a lot of time kinda like, uh, not only getting into the head of the, the AI and sort of being like, "Okay, well, why are they getting confused? You know, what am I sending that is confusing? How do I send more information in order to like produce a better result?" Uh, but also it really forces you to clarify your own expectations of... Like somewhere up here I have a, uh, a, a drag and drop list, you know, where you can drag list items between... And like I started working on this and started speccing it out. I was like, "Man, this is like actually like not only really hard to, um, like to produce a good result, but it's also like just really hard to describe." Is that like the failure was really on, on my end for just not knowing how to get the information in there because I didn't actually know how this thing should work. Um, but yeah. But, you know, I could figure it out. I, I have an environment in which to figure that out. It's fun. **Host** [40:59] That's amazing. Uh, I'm still processing. Oh, you have a picture of a, a- **Steve Ruiz** [41:04] Oh, yeah. All right. So this is, this is kind of annoying, like, uh, slightly obnoxious but, uh, during this ma-- Like, 'cause this thing went massively popular on, on Twitter. Uh, uh, yeah, like thousands of retweets type of level. Um, and there, there were some folks who like were sub-tweeting it about like, you know, "Get over it. It's just a wireframing or no-code tool" or something like that. Uh, one guy did say like, you know, "I prefer to, to wireframe like the old-fashioned way with pen and p-paper." And, uh, I was like, "Oh, yeah, no, that, that works too." Like this works with screenshots. I can just take this screenshot here of, you know, that the, the dude posted of, of the, uh, the, the drawing that he had made, you know. It's not even like a good photo. There's a pen, you know, across one of the screens, et cetera. But if you just give that with no other information, uh, to like as a prompt, uh, you get back a, a pretty good result. Like it, uh, loading right now, but, uh, I've done this before and yeah, like you get back um, you know, just from this like photo of a, of a piece of paper on the, the, the guy's desk, you know, you have a, um, like not, not completely arbitrary, uh, result. Like working website here, um, that was, that was, that was inferred from just that picture with no other input. Not, not even like titles or anything else. And of course it's like re-responsive and, and all this stuff. Uh- **Host** [42:39] Yeah **Steve Ruiz** [42:39] ... and so the idea of like, yeah, like our tools-- Yes, I've, I've worked really hard to make all of our ... shapes, you know, really good and our arrows obsessively good and, and all this stuff. But like, the fun of the infinite canvas in tldraw in particular, is that you could just dump like whatever you want onto the canvas: screenshots, text, images, other websites, uh, sticky notes, all that stuff. And, uh, the, the, the model, even f- as something that was in preview, like the very, very first sort of multimodal, um, model, uh, can do a really good job at just taking all that stuff as the input and, um, and yeah. Like, so we accidentally made a really, really good visual multimodal prompting, uh, uh, application environment or, or user experience environment. I'm not even sure what we're gonna call this thing. **Host** [43:38] It's a tldraw. Uh. **Steve Ruiz** [43:41] Ooh. **Host** [43:42] Uh, you, you also had a- **Steve Ruiz** [43:43] It's a tldraw **Host** [43:43] ... in our pre-show prep, you also talked about parallel prompting. Is that basically just, uh, prompting and then moving on to something else? Is, is that, is that what you're referring to? **Steve Ruiz** [43:50] Yeah, that's kind of what we did up here with the, uh, the stopwatches. The fact that we could get multiple prompts going at the same time and like arrange them spatially, uh, and, um, people have done this also with, with imagery, uh, to say, "Okay, well, here are... We're gonna use DALL-E, and we're gonna kind of like make a tree of prompts, uh, as you go, you know, different iterations based on, um, which... " Whatever. You make four iterations, you pick your favorite one, you keep going, kinda like what you do in Midjourney. But to have that spatial and to have that like, uh, arranged on a canvas so that it actually can make sense to you, and you can kinda look back and follow it, follow forward, that like... I don't know, like Whiteboard's infinite canvas stuff is just really good for a lot of things. Um, and, uh, so organizing like a whole bunch of different content that is, uh, irregular or ephemeral or, um, or, or has a kinda like ad hoc meaning configuration. Like, you know, things that are next to each other or things that are in a grid or in this case, you know, uh, just even what we're, what we have here for what we did with the stopwatch. Like there's an implicit meaning of like the, okay, the source is on the left, the r- result is on the right, and any further iterations are further on the right, right? Like there-- We, we didn't put that into a data model. We didn't structure that in any way. It doesn't actual- That meaning relationship doesn't really exist in any part of the product. It just exists to us because we, that we can make sense of it. Um, and w- yeah. So for, for this type of thing, not only is it cool that now a, a model can make sense of it as well, but, uh, but yeah, for, for organizing, uh, complex iterations of imagery, complex iterations of, of, of outputs, et cetera, like yeah, the canvas is a place. Uh, I really b- do believe that. **Host** [45:52] Yeah. Um, I mean, that's, that's, that's really incredible. Uh, it's also-- It's a little, you know, I, I think a, a few developers are kinda scared, uh, about, uh, you know, how much of this, uh, their jobs, uh, this does. Obviously, uh, there's a lot more that it can do. **Steve Ruiz** [46:08] Yeah. So the, the will this take my job story is, is interesting at this moment. **Host** [46:14] I, I mean, I'm not actually concerned, but I'm, I'm curious what, uh- **Steve Ruiz** [46:17] Yeah. **Host** [46:17] Um, I, I think this augments. Like, uh, my... Actually, my concern as a developer is that this is good, but not good enough, you know? **Steve Ruiz** [46:24] Mm. **Host** [46:24] Like, it's good for throwaway UI, um, but I... Would I actually export the code and take that code? Um, I don't know. Uh, it, it looks like, uh, your first MVP was just HTML files, which, you know, if it's a single HTML file, it can, it can have some JS and some CSS. **Steve Ruiz** [46:42] Mm-hmm. Mm-hmm. **Host** [46:43] Um, I saw some problems with layout, uh, in there, which I don't know- **Steve Ruiz** [46:46] Oh, for sure **Host** [46:47] ... how good it is at layouts. Uh, it's, it looks like you could just prompt it for Tailwind if you want Tailwinds. Uh, I assume it can generate React. Uh, I, I don't know. What, what are the limitations of this thing? **Steve Ruiz** [46:57] Well, there's, there's the limitations that are in that particular demo, which is that like, well, it couldn't do React because it needs to just be a single compiled thing. Uh, you can't do any multi-page stuff or, or anything like that. But that's more of like how we're structuring the, uh, the project rather than like a specific requirement of, of the project itself. The... I mean, we've talked about having... Well, there's, there's two kind of things. There's one is like how big is the input window and how big is the output window or something. **Host** [47:28] Ah. **Steve Ruiz** [47:28] Uh, so in theory, you could have, uh, the input be, "Here's a entire full stack React application, uh, together with all my UI and all this, all my components, et cetera, and here is a screenshot that I took of the landing page where the menu, uh, is in the wrong spot, you know. And I'm, I'm gonna annotate that with some arrows and some text in order to say like, 'Here's what I, where I want it to be,' or, 'Here's what I want,' et cetera." And for the output to be, um, you know, a diff that I can apply to my code base. Like produce the, the, the, the Git, the, like, basically like produce the commit, um, that would change this and have that commit be against multiple files and, and, and et cetera, um, in order to, to have potentially like a, a solution that is just, uh, uh, like ready to go, applicable, like a patch or, or, or a PR that you can make. Um, there really isn't any, any, any limit in that. And, uh, we've seen with Copilot, et cetera. The challenge is more on the input side than the, the output side. Like, uh, like absolutely, you could figure out a way for this thing to spit out like a, a working iOS app or something like that. Uh, the question is like how do you Tell it what you want and, and how do you iterate when it gets it wrong? And, and just doing zero shot, zero shot, zero shot is, like, really a frustrating, um, process. Uh, but if you do have a way of, of iterating, if you do have a way of kinda like step-by-step moving towards the, the solution that you want and kinda like, you know, getting it into like, okay, well, this is good, but it's not great, could be better, et cetera, um, that's, uh, that's how you actually make that type of complex output more practical or more realistic, um, is that you probably won't get-- ever get the prompt just right, even if you have like a really, really, really good, you know, three generations from now agent. Like, you, you still have to put that information in, um- **Host** [49:37] Yeah **Steve Ruiz** [49:38] ... but you're never gonna put the in- all of the information in the first time. You need to be able to, to iterate on it. And so with visual stuff, uh, I feel like the canvas, like what we were looking at, um, that's part of what it unlocks is that, like, space of iteration. That space of, uh, um, you have a way of marking up the result and using that as the new prompt. Um, and that's, that's kinda new. **Host** [50:02] Yeah. Yeah. Um, i-i-- multimodal prompting is, is such a brilliant concept that, uh, you know, I think it's, uh, it's gonna be a norm for, for, for some things. Um, i-in my mind, you demonstrated, you know, uh, I would-- Coming from Photoshop, there's this concept of layers. Um- **Steve Ruiz** [50:21] Mm-hmm **Host** [50:21] ... you can, you can kinda simulate layers in, in tldraw. Uh, and I see like a emergent property of layers in, in this kind of prompting, which is there's the UI layer, and then there's the state chart layer. Um- **Steve Ruiz** [50:34] Mm **Host** [50:34] ... and those two things seem, like, pretty useful in, in, uh, spec-- specifying your prompt. I was just wondering if you've, if you thought a, a little bit about, like, other dimensions or other layers that would be useful in multimodal prompts. **Steve Ruiz** [50:47] Yeah. So, uh, one thing that we've done is to bring in screenshots of other apps. Like, "Here's stripe.com. Like, make it look like Stripe," you know? Uh, so just sorta like, like... Or like, "Here's linear.com. Like, let's, let's do it this way." Uh- **Host** [51:07] Make my dev tools website, uh, or make- **Steve Ruiz** [51:09] Exactly. **Host** [51:10] ... pop. Uh, you, you should just, you should you should just, like, give it this item and ask it to make pop instead of make real. **Steve Ruiz** [51:17] Yeah, exactly. Make it, make it more, make it more, make it more pop. Uh, so there, there's the idea of, like, bringing in style as, like, a... as another, um, another in-- like, part of the input. Uh, flow charts are, are absolutely useful. Uh, I mean, this is-- It really just boils down to, like, what would you really give a developer who you are working completely asynchronous with? You know, if you had to, uh, spec out a project, uh, and put the... print it out on paper and mail it to a developer, and they were gonna mail back a, a disk with an HTML file on it, um, you know, like, what would you send, uh, if you were sending this to the moon or something? Uh, and so yeah. Definitely, like, descriptions of how the state should operate and specs on that. Uh, we've, um... What else have we done? We've done flow charts. We've done screenshots of other apps. We've, uh... I think we've, we've even just pasted in code. Like, like, "Here's a whole bunch of JSON that you can use," um, and have it just read that as the, as the input data. Uh, we can-- You could point it at specific endpoints. You can say like, "I want you to hit this endpoint and then display the results, you know, as, uh, as cards or as items or something like that." Uh, not-- I mean, you don't even have to, like, wire this up. It's not like Retool or anything where you, you have to register that n- or, you know, it's not built into the tool. You just- **Host** [52:53] From an endpoint. **Steve Ruiz** [52:54] Oh, yeah. Yeah, yeah, yeah. I'm trying to think of, uh, what a good demo endpoint would be. We could-- Maybe we could do one more, uh, more te- What is it? Like dog.co? Is that- **Host** [53:05] Yeah **Steve Ruiz** [53:05] ... is that still around? **Host** [53:06] Dog.co is, is a good demo. **Steve Ruiz** [53:06] Yeah? **Host** [53:07] Yeah. I've used that one. **Steve Ruiz** [53:08] Um, I mean, this, this might be, this might be kinda like the box with the word calculator. Like, it might just know because it's probably been in a bunch of tutorials. **Host** [53:17] Oh, it's in the big training set, yeah. **Steve Ruiz** [53:20] Uh- **Host** [53:20] You're not, you're not sharing- **Steve Ruiz** [53:20] But, but, uh, you know what? We'll, we'll do it anyway. We'll, uh... I'll, I'll share it, and, uh, we'll, we'll try. Uh- **Host** [53:26] So for those who don't know, dog.co is, is one of those, like, uh, demo APIs that, uh, you just set up- **Steve Ruiz** [53:32] Yeah **Host** [53:33] ... just because it's not offensive and, uh- **Steve Ruiz** [53:36] Yeah. Exactly **Host** [53:36] ... it's got useful dogs. **Steve Ruiz** [53:38] So you can, you can get dog.co. **Host** [53:39] Everyone likes dogs. **Steve Ruiz** [53:40] Yeah. **Host** [53:41] Yeah. **Steve Ruiz** [53:41] Um, and I'll try and think if there's a, there's a more complicated endpoint that we could hit. Uh, maybe, maybe while I'm doing this- **Host** [53:47] Uh, I mean, this is good **Steve Ruiz** [53:47] ... if you wanna, if you wanna think about it. **Host** [53:48] It gives me more ideas of... Yeah. **Steve Ruiz** [53:50] Yeah. **Host** [53:50] It gives me more ideas of what to prompt and what, what can go in. Uh- **Steve Ruiz** [53:54] Yeah **Host** [53:54] ... I definitely didn't think about hitting endpoints just 'cause I-- it's just not in any of the demos I've seen. ### Generative Experiments **Steve Ruiz** [54:00] Yeah. Uh, but, but it works. Um, let me see. I'll, I'll have a big button down here. Uh, show me a dog. Okay. So that's gonna be our Show Me a Dog button. Uh, this should be a picture of a dog. **Host** [54:22] Oh, that's a great dog. **Steve Ruiz** [54:23] Uh- **Host** [54:23] No, that's a good- **Steve Ruiz** [54:24] Thank you. **Host** [54:25] Oh. Yep. Yep. **Steve Ruiz** [54:28] Yeah. Uh, and uh, and then we'll, we'll do some annotations here. We'll say like, uh, uh, okay, when, when this is clicked, get a new dog. Um- **Host** [54:44] Yeah. This is, uh, perfect arrows coming in. **Steve Ruiz** [54:48] Yeah, exactly. When clicked, get a new dog from dog from... I'll just paste in this. Um- **Host** [54:56] Yeah **Steve Ruiz** [54:57] ... and put the result in the image. Okay. So it, it's more of a, more of an instruction than you would normally, you know, I don't know. **Host** [55:09] Yeah. One thing that- **Steve Ruiz** [55:09] But it's, uh- **Host** [55:10] ... it's gonna have to guess is, uh, you know, the, the response format, right? 'Cause it could be anything. **Steve Ruiz** [55:16] This is true. Let's see if it works. Well- **Host** [55:20] Nope. **Steve Ruiz** [55:20] Hey. All right. **Host** [55:21] Oh. **Steve Ruiz** [55:21] Well, that one worked. Yeah. And let's see if it a- **Host** [55:25] Yeah, go on **Steve Ruiz** [55:25] ... if it actually hit it, uh, hit the, uh, the right, um, the endpoint in the, in the right way. So, um, dog button. Yeah, okay, it hit the right, right endpoint. JSON, dog image, uh, and then it put it in. So there you go. Uh, you have yourself a, a JavaScript tutorial in a box ready to go. And I think, like, I mean, we probably wouldn't do this on camera, but, like, you can say, you know, like, like, use the auth token, you know, whatever, and like, you know, go, like, really get, you know- **Host** [56:03] Right **Steve Ruiz** [56:03] ... real data back from this thing. Um- **Host** [56:06] Yeah **Steve Ruiz** [56:06] ... there's not- no reason why it wouldn't be able to do that. **Host** [56:09] Yeah. **Steve Ruiz** [56:09] Uh, so yeah. **Host** [56:11] You're, you're kind of relying- **Steve Ruiz** [56:12] It's really the only- **Host** [56:12] ... on the OCR. Yeah. Yeah. **Steve Ruiz** [56:14] Yeah. Well, not really, because, a- again, what, inside of the prompt for this, we do give it, like, an array of all the text that you've put in. **Host** [56:26] Oh. **Steve Ruiz** [56:26] We, we say like, "Look, I know your vision isn't so good, or you have a hard time reading text sometimes when it's small." Uh, 'cause what it... Like, the input that, that you get is pretty wild. It's like it takes this as a PNG, uh, and then it like, um... I can't do this in tldraw, but it, it resizes it. It, it squishes it into a, a 512-by-512 image- **Host** [56:49] Yeah **Steve Ruiz** [56:49] ... or something like this. **Host** [56:50] And tiles it, yeah. **Steve Ruiz** [56:51] Uh, and then ti... Yeah, so it's like, um, it-- Text especially can get kinda, like, chunked up, especially if it's small. So we, we send those strings separately, uh, so that it can kind of reassemble, um, anything that it can't read, uh, right off the bat. Yeah, it's, it's, this is a weird future that we've, uh, we've found ourselves in. It's pretty cool. **Host** [57:18] It's pretty cool. Yeah, I mean, you know, one, one layer I a-automatically think of is, is back-end, right? Like, uh, as someone who- **Steve Ruiz** [57:24] Mm **Host** [57:24] ... has, uh, works at AWS, um, I see a lot of systems diagrams, like cloud diagrams, um, entity relationship diagrams for database schema. So I wonder if, like, anyone's tackled extending this to back-end, and then obviously the next level from that is, uh, full stack apps where you have back-end and front-end. **Steve Ruiz** [57:43] Yeah. Um, short answer is yes. Uh, there, there's someone on Twitter that was using this to generate, um, yeah, like he was doing like flowcharts. I'm not a back-end guy, so I don't actually know exactly what the output was. **Host** [57:58] Okay. **Steve Ruiz** [57:58] Um, but I, I believe it was like a, like a configuration script for AWS, um, that was built off of this like... I think he just copy and pasted a, a, a diagram that he had made in tldraw anyway and said, "Okay, well, let's throw this at this thing and see what it comes up with." Tweaking the prompt to say like you, um, rather than building single-page websites, you just return the JSON, you know, description of this configuration or something like that, or return a script that would set this up. Um, you could tweak it to say like, "Here are like, yeah, all the entity relationships between, um, uh, different tables or, or items in tables, uh, and, and give me the, give me back the SQL, you know, con-- like, you know, initialization or something so that would make all these tables and, um, and set up these relationships." Uh, yeah. It's just again, like, the hard part is getting that information in, but we... I don't know. Pictures are really good. Um, and yeah. **Host** [59:05] They make- **Steve Ruiz** [59:05] If you can tell it that way, then it works **Host** [59:06] ... it works. **Steve Ruiz** [59:08] Yeah. **Host** [59:08] Yeah. Awesome. Um, you are also part of, uh, so you were one of two, one of what I think about, uh, multimodal, um, viral hits in November. Uh, the other one also you had a pl- a part to play in, which is the local consistency models, uh- **Steve Ruiz** [59:25] Yeah **Host** [59:26] ... trends, uh, where I think you worked with Felt. **Steve Ruiz** [59:29] Yeah. So we, uh... And actually, I do have something to show here. We, we actually have a couple of things to show here. Uh- **Host** [59:36] All right **Steve Ruiz** [59:36] ... we connected with Felt because they used, um, tldraw to, to create a demo for, uh, for their, their cons- like, uh, LCM, LCM, right? **Host** [59:49] Yeah. **Steve Ruiz** [59:50] Latency consis- Yeah. Um, but we took that, and we, we made, uh, drawfast.tldraw.com, which is, uh, basically you get a, these like shapes, these little draw fast shapes, um, and it puts the result, basically go-- grabs that new image and puts it right next to it. And these are, um, these are extremely fast. So as I'm moving things, you should see- **Host** [1:00:19] Oh, nice **Steve Ruiz** [1:00:19] ... the image, um, updating as well. Uh- **Host** [1:00:23] Goodness **Steve Ruiz** [1:00:23] ... and I think it, I think this was originally not a goat, you know, whatever. This is a, um, a, a wise princess. I don't know. I, I play this more with my, uh, my daughter than anything else. I like seeing what this looks like. **Host** [1:00:42] Yeah. She has something. **Steve Ruiz** [1:00:42] You know, and, um... Oh, my gosh, she, she does. And actually, we had a, a lot of, uh, lot of folks on, uh, Twitter, um, being like, "This is, this is not good." Like, whatever. Like, 'cause I had a video of whatever, my, my daughter drawing and, you know, she made this awesome drawing of a, a mermaid, and we turned it into like this really anonymous- ... crappy version of like an illustration of a mermaid. And, uh, they're like, "No, no, the children's drawing is much more interesting." And I'm like, "Yeah, yeah, yeah, come on. What, who, who cares? Of course it is." But, like, this is, uh, you know, this is, this is fun. So- **Host** [1:01:20] Yeah **Steve Ruiz** [1:01:20] ... like- **Host** [1:01:20] I, I do think you, you might be doing- **Steve Ruiz** [1:01:21] It, it, it gets pretty wild **Host** [1:01:23] ... animation. **Steve Ruiz** [1:01:24] Yeah. **Host** [1:01:24] Like some kind of, like you could make, uh, some kind of... It's, it's a, this is almost like stop-motion film. Um- **Steve Ruiz** [1:01:31] Yeah. **Host** [1:01:31] Yeah. I mean- **Steve Ruiz** [1:01:34] Yeah, we, uh- **Host** [1:01:34] ... we just, we, we need to do more work on the consistency, but, like, this is getting there. **Steve Ruiz** [1:01:38] Yeah. Uh, it is a car driving through a busy market, marketplace. Uh, the fun is that, like, you end up, uh, after playing this, with this for a little while, you end up like, uh, getting really into the particularities of the input. Like the, uh, like what can you do with a design tool? Okay, you can move things around, right? I can grab some of these and move them around. Um, and like, oh yeah, there's a highlighter here too, so we could like, we could do some highlighting, you know, that'll, that'll do stuff. Uh, and then you kinda... Like we, we couldn't help ourselves. We started making these like stories. So, uh, one thing you can do is kind of click this little button, and that'll make the, the drawing and the- **Host** [1:02:25] Oh, yeah **Steve Ruiz** [1:02:25] ... result kind of on top of each other. All right. Well, then I'll move on to the other one that we, uh, that we released earlier today- **Host** [1:02:32] Yeah **Steve Ruiz** [1:02:32] ... which is called lens.tldraw.com. So that was drawfast.tldraw.com. Uh, and again, this is probably not making a good podcast audio, but it is, uh, the- **Host** [1:02:42] It's great video **Steve Ruiz** [1:02:42] ... the image updates as soon as possible based on what the input drawing is. Um- **Host** [1:02:48] Yeah **Steve Ruiz** [1:02:48] ... and it is pretty hypnotic. So the, uh, this one's a little riskier 'cause it's live. So this, this is a... We took a project called Together, which is a vertically scrolling infinite drawing collaborative experience. Uh, it's a little bit like a chat room. As you're drawing, everything's just sort of moving up. **Host** [1:03:07] Yeah. **Steve Ruiz** [1:03:07] And it just disappears off the top of the screen, never to be seen again. So it's kind of just fun to play with. Um- **Host** [1:03:14] By, by the way- **Steve Ruiz** [1:03:14] And- **Host** [1:03:14] ... one of the most magical chat experiences I ever had was with you. Uh, I think you were like with your daughter or something, and I was, I was, whatever, showing off together. And you started writing, I started writing, and we had chat, uh, on together.tldraw.com. **Steve Ruiz** [1:03:28] Yeah. Yeah. **Host** [1:03:28] I was, I was like, "Wow, what is this?" **Steve Ruiz** [1:03:31] It's, it's super cool. Like I, you know, there's inevitably someone will write like, you know, "Where are you from?" And, uh, and everyone's like chiming in and, and talking about. Yeah, you, you- **Host** [1:03:40] It's- **Steve Ruiz** [1:03:40] ... or like- **Host** [1:03:41] Yeah **Steve Ruiz** [1:03:41] ... "This is cool. Where..." You know? Yeah, you can chat. Um- **Host** [1:03:45] Yeah. **Steve Ruiz** [1:03:46] So what I, I'll describe what's on the screen now, which is basically, uh, we've-- we're taking like a screenshot of the center, uh, like a square out of the center of this chaotic, vertically scrolling chat experience, and we're sending that to the, to the LCM and putting back the image based on like a prompt, uh, like, you know, desert scene or busy marketplace or, uh, futuristic cityscape or something like that. And so it is updating like, you know, 10 times a second as, uh, as we go. But yeah. **Host** [1:04:20] I would say like it's updating surprisingly quickly, like 10 times, 10 frames per second. **Steve Ruiz** [1:04:24] Yeah, here we go. **Host** [1:04:26] Yeah. **Steve Ruiz** [1:04:27] Yeah. Uh, no, I think it's now like to 32 milliseconds basically, uh, as you go. And, um, so if I draw like a big orange thing down here, it's gonna kinda show up into the drawing. Um, maybe I'll, I'll do a big black one so you can see better. Uh, like it just sort of becomes part of the, the input to this prompt. Um, and it is extremely hypnotic. Uh, this is again, like lens.tldraw.com. Um, this isn't live, so no one should come on here and say, "Hello Steve" or, or anything like that. But it is, uh- **Host** [1:05:04] No. Yeah, no, please don't. **Steve Ruiz** [1:05:05] It is... Yeah, it's like this like slow-moving, um, collaborative kinda like hallucination experience. And it just never ends. **Host** [1:05:16] It's fantastic. **Steve Ruiz** [1:05:16] I mean, yeah. I'm- **Host** [1:05:17] Yeah **Steve Ruiz** [1:05:17] ... I'm probably gonna be funding PHAL completely for the fir- next, uh, you know- ... their, their Series A or something like that. But the, uh, yeah. Are you on here, uh, Sean? **Host** [1:05:30] Uh, I, I'm not, but maybe I should. **Steve Ruiz** [1:05:32] Okay. Well, whatever. **Host** [1:05:34] Yeah. Uh, I, I, I have like, I don't know, I have a healthy respect for like the, the amount of processing that must be going on behind these things. Um, I, I just- **Steve Ruiz** [1:05:45] Uh. Uh, yeah. Well, what's funny is that, uh- **Host** [1:05:51] It's me. I see you, I see you **Steve Ruiz** [1:05:52] ... oh, cool. Perfect. Someone's doing a- **Host** [1:05:53] Oh, it looks like I can't really draw- **Steve Ruiz** [1:05:55] ... spandex **Host** [1:05:55] ... the way I normally draw on Tldraw. You, you, you blobbed it somehow. **Steve Ruiz** [1:05:59] Yeah. It's, everything's a little bit bigger. Um- **Host** [1:06:02] drew a little smiley face. **Steve Ruiz** [1:06:03] When there's 36 people, uh, it's, it's kinda slow as well. So, uh, yeah. We, um... What, what's funny is that like, yeah, we're using like Cloudflare workers to do the, uh, the, the, the updates and the CRDTs to do the collaboration and, you know, all this like whatever LCM models to, to, to, to populate this image or create this image. **Host** [1:06:28] Yeah. **Steve Ruiz** [1:06:28] But, um, there's also a laptop in my living room right now, uh, that is doing the actual screenshotting, uh, and sending that up. And so there's a big note that I had to write, you know, for my family to say like, "Don't turn off this laptop. Don't close this laptop because, uh, this needs to be on in order for this thing to work." Uh, and yeah. So, you know, we'll-- no matter how good our tech stack gets, we'll always come back to, uh, some, some laptop stuck in the corner that can't possibly be turned off. Uh- ... it's pretty fun. **Host** [1:07:02] Yeah. I- **Steve Ruiz** [1:07:02] Yeah **Host** [1:07:02] ... I've heard of major businesses being run that way. Uh- **Steve Ruiz** [1:07:05] Yeah, exactly **Host** [1:07:05] ... that's, that's a risk **Steve Ruiz** [1:07:06] ... there's like a Raspberry Pi in the closet. Yeah **Host** [1:07:09] I, I-- You know, it's weird 'cause it, it's really funny 'cause like, you, you know, you, you are inventing your own art form. Uh, this is, uh, fine art, uh, you know, going back to your degree. Um, it's just a different level. **Steve Ruiz** [1:07:19] It's definitely a, uh... You know, it, it's funny because, like, the output of this, like, while it is like a visual output, the output, like, doesn't actually matter. Like, it's gone in, in 16 milliseconds, and it's, it's not coming back. Uh, and I think with all this AI stuff right now, just where we are with it and just how- **Host** [1:07:40] Mm **Steve Ruiz** [1:07:40] ... completely unknown it is in terms of, like, where is this useful. Uh, like, the best thing that you can get out of this is, like, the experience. Uh, and so I think of this much more as like, you know, the thing that people will walk away from, from playing with, like, lens.tldraw.com, uh, should be more of, like, that experience of, of having interacted with this thing or interacted with it, you know, among, with others, uh, rather than like, "Oh, there's... It made my favorite image," or something like that. So I think the, uh... Yeah, I, I don't know. As a, a-a-as a former image maker, like, the idea of having, um, having like, you know, a, a aesthetic experience where the image is, uh, a major part, but it's, it's not necessarily like the, the important part, uh, or any one of these images isn't the important part. I don't know. There, there's some like... There's some, uh, there's something, something new feeling about this. Kind of fun. Certainly- **Host** [1:08:45] Yes **Steve Ruiz** [1:08:45] ... I, I wish I was, uh, could do a, a big critique with all the new media artist people, um, about this and about like what, you know, where does this, where does this fit into the sort of the, uh... Yeah. **Host** [1:09:00] Yeah. **Steve Ruiz** [1:09:00] Like other people's work, etc. **Host** [1:09:01] Well, that's for, that's for them to, to write and, you know, for you to build. Uh, and I think that's a, that's a... You know, I, I would encourage you to keep building there 'cause, uh- **Steve Ruiz** [1:09:09] Yeah **Host** [1:09:09] ... you're definitely not done explo- uh, with your explorations. Um- **Steve Ruiz** [1:09:13] Oh, thank you. **Host** [1:09:15] Cool. Yeah. Well, so I can sort of round it out by sort of looking towards the future. Uh, you hinted a little bit, uh, you're working towards tldraw 2.0. Um, w- so first of all, actually, w- so, uh, it seems like you're very focused on the core mission of Canvas. ### Tldraw 2.0 **Host** [1:09:32] Um- **Steve Ruiz** [1:09:33] Hmm **Host** [1:09:33] ... and the AI stuff is, is a side project for now. Um, why, why not pursue it as like a, a full- Why not pivot and, like, be an AI company, right? Like that- **Steve Ruiz** [1:09:43] Yeah. Yeah, I guess I- **Host** [1:09:44] That's, that's... I'm sure you've had a lot of those questions. **Steve Ruiz** [1:09:46] Yeah. I mean, when you, when you get something as viral as, as tldraw got, like I, I think I've talked to everyone, um, certainly every, every investor. Uh, and, um, okay, so I guess the way that, the way that I... Yes, we, we probably could on for something like Together or that Draw Fast thing, uh, make a, a tiny little SaaS app, you know, give me $10 a month, play with this thing, and, uh, you know, could make a, make a good... We could go in that direction. Um, there's not much of a moat around any of this stuff, and, and we're seeing that just in, uh, you know, I don't know, Gemini is gonna come out in a couple of days, weeks, or whatever. Um, and if it's better, then people are just gonna use that until the next better thing comes along. Like, it's not... There's not a lot of like unique, uh... There's n- there's nothing really defensible about, like, hey, it's an, it's a drawing app plus an LCM-like model, uh, because there's gonna be a lot of those models, and there's gonna be a lot of drawing apps. The, the thing that I think is really unique for tldraw, the thing that we have added that is not easily created, is the canvas itself. Is that, like, this, uh, you know, web-based, uh, hackable, extendable, uh, super refined, um, interactions and, and all that stuff. Like, you know, all the thousand table stakes features that, that drive people nuts when building something like this, like, they're all there, they're all good. Day one, you could build a really great experience, uh, whether it's AI driven or not, like using tldraw, um, in a way that it's just not practical to do if you're building it yourself. And especially if you're not doing like graphic stuff, there's really not that much else out there, uh, oriented towards, towards this type of thing. Um, and I think in a world where, like, these type of experiences are gonna... Or not these experiences, these types of AI-driven capabilities are just gonna keep coming out faster and faster. And like, you know, I don't know, next, next year is gonna be wild. Like, every month there's gonna be some new, uh, you know, capability or something. The, the thing that I would want to see, both just me as a, as, as a person and as me as having built the business that I've built, uh, is for tldraw to sort of become the, the place where some of this prompting, some of these i-ideas are explored. Um, even if we decided to, okay, we're g-just gonna close everything up, we're gonna build a product b-based on this, that... And maybe it's a great product, but it's, it would only be one kind of, one direction, one ray kind of into this infinite space of, of, of possibility. Uh, and that, that could be successful, good, but, like, the idea of building the, um... I mean, we've built the, the sort of the direct manipulat-manipulation core, but there are so many, even like AI specific APIs that we could build around tl-tldraw for having, you know, like a virtual collaborator, uh, or, or, or working with images in a, in a more, more rich way. Like, there's just so much that we could build in order to make this the best possible place to explore, not just one direction, but like, you know, many, many, many directions. Um, and I think that That narrative gets me out of bed in the morning in, in a much more, uh... That, that gets me much more excited. Um, and I think we're also just, like, the team that we have and, and the, the tech that we have and the skills that we have, we're more the team to build that rather than, like, to, um, become like a SaaS product, uh, company. I'm not saying we'll never do, like, a, you know, pay us 10 bucks a month and we can-- you can play with our magic toy. Um, but, uh, primarily my goal is to make, uh, Tldraw the either the place to explore these different models or you might, you might think of it as like, um, the battleground on which the, the winners will be kind of identified. Um, like right now we're using OpenAI for the Make Real thing. Um, uh, maybe next week we'll be using, uh, Gemini, and now it's, now it's a question of, okay, well, now we have an environment in which to compare these two models, um, with the same input and a very advanced form of input. Uh, but yeah, like let's see which one does better. Now nothing would make me happier than to be sort of like the, the battlefield, um, for multimodal prompting and multimodal, uh, AI experience. **Host** [1:14:32] Yeah. I should also shout out BakLLaVA as, uh, one-- the open source, um, vision and, um, multimodal model. Um- **Steve Ruiz** [1:14:40] Sick. **Host** [1:14:40] Yeah, I mean, like, uh, so, so I fully understand you wanna, you wanna own the light cone of, of multimodal prompting. Uh, I think that-that'll probably be the title of the episode. Um- **Steve Ruiz** [1:14:51] I like it. **Host** [1:14:51] What, what, uh, what's coming up for Tldraw 2.0? **Steve Ruiz** [1:14:55] So really the Tldraw that you are using now and that I'm using are, is basically 2.0. It's been in pre-release for a long time. Uh, really the only change that's gonna happen once we launch it is, uh, we're gonna start selling commercial licenses for it. Uh, so if you are using Tldraw in a commercial product or if you want to, then, um, you know, if it's... If you're funded or if you have revenue, then, you know, you'll buy a license and I'll, um, add you to our, our special list of customers. Um, so yeah, it's mostly just go to market and the, the necessary changes around there. There will be some kinda fun changes, secret saucy changes at that launch, but nothing substantial, nothing breaking. Uh, we've put a lot of effort in the last... Like it's crazy that we've only had it open source since May of this year, uh, this, this new version, right? And yeah, it's-- we've been very busy since then. But it is, uh, it's stable, it's robust. We put it through, you know, a lot of usage and caught a lot of the, the issues, so it's absolutely ready to go. Um, but I have, uh, one or two conversations with my lawyer before we, uh, we turn, turn over the license and start, start moving it that way. **Host** [1:16:10] Gotcha. Um, and then, uh, maybe, uh, I think if I could get your commentary, uh, before we, we close on, on, um, just the, the competition out there. Like, um, uh, so you are not... You are not the only sort of canvas tool. ### Competition **Host** [1:16:22] I, I think I get now that... I, I was gonna ask about like Figma, FigJam, and like they have some AI thing that they're also doing. **Steve Ruiz** [1:16:29] Mm-hmm. **Host** [1:16:30] Adobe is also working on similar things. Canvas is also working on similar things. But they're all individual point solutions, whereas you're more the open source, uh- **Steve Ruiz** [1:16:38] Yeah **Host** [1:16:38] ... sort of, uh, canvas to power all of them. Um, yeah, I feel like it's just Excalidraw that's like the other alternative, uh- **Steve Ruiz** [1:16:45] Yeah **Host** [1:16:46] ... that remains. **Steve Ruiz** [1:16:48] I think Excali-- And I like Ex-Excalidraw a lot. I, I contributed there and, you know, we, we retweet each other and tease each other on, on Twitter and stuff. Uh- And, uh, early on I was copying features from them. Now they're copying features from me. So I, you know. **Host** [1:17:03] That's how it is. **Steve Ruiz** [1:17:04] But no, it's, uh, it's... Like the collaboration space is so-- has so many dominant players like, um, that I, uh, I think me and Excalidraw are, are tiny within that. Um, yeah. Well, I, I... There's two things. One is that we made this very strange bet on using, uh, kind of a web canvas, that our canvas is not like an HTML element or HTML canvas element. It's like normal React components all the way down. So if you wanted to add something interactive and have that participate in the c- sort of space of the canvas, uh, the way that we were doing our, our, um, iframes, kinda like being able to write on top of an iframe, you can't do that in Excalidraw. You can't do that anywhere. Um, that is like a very strange tech choice that we made around, uh, Tldraw that is, you know, finding its home in, uh, in a, in a few different ways. Most of the people who pick Tldraw and approach me, like the inbound that I get are folks for whom that's like the killer feature, um, to be able to, to put interactive widgets on the canvas using just React. Uh- **Host** [1:18:18] Yeah. **Steve Ruiz** [1:18:19] The... Yeah, I, I'll just kind of repeat the same point that you, you, you kind of suggested, which is that like, no matter how good Figma's, uh, like AI solution is, and I hope it's great because I love Figma and I use it, um, it's not gonna solve every possible problem in this space. It's not even gonna like touch- You know, like, like it can't. Like none of these things. And I mean, I already had identified like, okay, uh, there was a point where like any Kanban board was like, uh, uh, was Trello, right? When you, when you talked about Kanban boards, you were talking about Trello. Kanban boards are in every productivity app now. Uh, I think the same thing is gonna happen with collaborative whiteboards. It's like people like them. Uh, I'm making it easy. Uh, people are already doing it even without Tldraw when it's hard. Like, uh, like yeah, that's gonna become a kind of a commodity user experience in, in a lot of different products. Um, probably- You know, g-give me a diagram from a text prompt. Like, yeah, that is probably gonna be a commodity too. Give me an image from a text prompt. Like, yeah, that's just gonna be everywhere. We're just gonna assume that that's, you know, it's like adding a GIF to a, to a chat or something. That... There's no moat there. Um, I do hope that Figma has an amazing AI integration, but I think the thing that it will help you do is use Figma. Um, like, it like, generating an image won't be super useful, but, like, generating it, uh, extend, uh, auto-complete this, uh, this, uh, design w- absolutely would be and, and I hope that they launch something amazing there. Uh, but yeah, there's, uh, I-- like I said, there's just a million different directions that this stuff could go in. Uh, the canvas is just, like, a input device. It's, it, it allows a certain type of user experience, um, and that's certainly not limited to design. That's not limited to whiteboarding. It's not limited to collaboration or anything like that. So, um, yeah, my hope is that there are those, like, kinda 10,000 products that, that could be made with what we're making. Uh, yeah. **Host** [1:20:32] Yeah. That's, that's a really, uh, great mission, and, uh, I, I see why you're so passionate about it. You have the right team for it. Um, okay. Uh, uh, you know, a couple lightning round questions, um, uh- ### Lightning Round **Steve Ruiz** [1:20:43] Sure **Host** [1:20:43] ... one which is, like, if you had some AI capability that you would wish for that you don't have yet, what would it be? **Steve Ruiz** [1:20:52] Oh, that's a really good question. Uh- **Host** [1:20:57] Helps people to do some research. **Steve Ruiz** [1:20:59] I think probably related to... It's not quite a CRM, but, like, a, a human, uh, just normal relationship management. Um, this is something that I've never had a problem with u-until, uh, until I had a startup, actually, where there's just a lot more people involved in my life, and it's, uh, um, it's hard to keep up with them all. And I think this is probably something that, like, an EA, like, kinda does of saying, like, "Hey, there's a birthday coming up," or something like that. Um, but also just, you know, identifying opportunities to, to work together, uh, to, to connect or, you know, who's an expert on this thing that I'm working on. Like, that doesn't always occur to me. Um, and I think there could be-- I don't know. I think the, the, the value of your network, um, that even if you're good at that, you're probably only scratching the surface of, like, you know, how, how you could be helping the people around you and how they could be helping you based on, like, the specific context of, like, what you're working on and the problems on your table today. **Host** [1:22:06] Yeah. I've also wanted to build a CRM on top of Twitter, uh, because- **Steve Ruiz** [1:22:10] Yeah. **Host** [1:22:10] ... you have all the info there about what people working on, your past conversations with each other, um, and your shared interests, you know? Like, and it should be able to search-- Y-you should be, like, at bare minimum search it, but to proactively suggest is, uh, the next layer. Um- **Steve Ruiz** [1:22:24] Yeah **Host** [1:22:24] ... and I guess AI chief of staff, AI executive assistant. **Steve Ruiz** [1:22:28] Yeah. Exactly. Yeah. **Host** [1:22:30] You know, something like that. **Steve Ruiz** [1:22:30] Yeah. **Host** [1:22:31] Understood. Yeah, yeah. Um, I, I think, like, some people are working on that, but the, the problem is so big that they're, they're working on, like, the automation piece. So, like, Lindy I had at my conference, um, where they're, they're like it's, it's, it's a virtual assistant you can, uh, trigger on your desktop or via email. Um, and it's, it w- mostly deals with scheduling, um, but also helps you do a little bit of research. Um, so that, yeah, I think the agents field will progress there. We might take 10 years to do it. **Steve Ruiz** [1:23:00] Yeah. Eh, I can wait. **Host** [1:23:02] Uh- **Steve Ruiz** [1:23:02] It's all good. **Host** [1:23:03] Yeah. And then, and then finally, um, uh, advice for founders. Like, what, what has helped you the most, uh, as, as a founder? You know, you're two years into your journey. **Steve Ruiz** [1:23:12] Yeah. I think the, uh... So this, this kinda comes a little bit out of, uh, like, what you learn in art school, uh, type of thing. Um- **Host** [1:23:24] Really? **Steve Ruiz** [1:23:24] ... but yeah. Uh, but one thing is that, that basically, like, uh, when you're a studio artist or you're i-in a studio, whatever, there's no external constraints. You just kind of are running on, "Well, what do I feel like working on?" Uh, and the further you get away from, like, what do I feel like working on, kinda like the worse your work becomes. So having, like, a really good, uh, feeling for that sort of desire and, and being able to respect and follow that, that desire as, like, a, um... Because it's not arbitrary, is that, like, if, if you really, really feel like working on, um, on, on, on a thing, like, it-- that might be the sort of the, the tip of a very complex, uh, iceberg of analysis of, like, the field or, like, what people are talking about or, or something, uh, that you-- directions and market or something like that. Like, I don't know. I think with with tldraw and with as, as a founder on this, um, the thing that I've tried to do and that I've tried to preserve is, like, the, uh, being able to prioritize based on, like, what is most interesting right now. Uh, and that is, that is true for, uh, what code we write and, like, what features we work on. That's true for, like, which partners we, you know, we spend time with in terms of who's using tldraw, uh, the types of problems that we want to solve. Like, using your own sort of sense of what's interesting as a filter, uh, and what you wanna work on, like, what sounds like a fun thing to work on right now as a filter is not, is not naive. Um, and it, it can be kind of part of your, your secret sauce, and I think it's, uh, I think a lot of early founders are encouraged against that and to, to be working backwards from a certain outcome and all that. Um, and yeah, you, you do have to, have to do that. You have to put that into the, um, into the mix as well, but, um, you know, be sure that you're, you're picking the best parts, um, out of that mix. I don't know, the parts that you wanna work on. **Host** [1:25:28] Yeah. Well, I mean, what's the point of doing this if you don't have as, uh, have some fun- **Steve Ruiz** [1:25:32] Yeah **Host** [1:25:32] ... indulging curiosity? **Steve Ruiz** [1:25:34] The worst case- **Host** [1:25:35] Which is what happened to you with Mark Zuckerberg **Steve Ruiz** [1:25:35] ... you'll, you'll build something that you love, yeah. Yeah, exactly. And, uh, good things can come out. Good things can absolutely come out of like, uh, chasing, uh- **Host** [1:25:42] You had an 8,000% increase- **Steve Ruiz** [1:25:44] ... chasing interest **Host** [1:25:44] ... in your followers or something. **Steve Ruiz** [1:25:45] Dude, not... Should I put this on camera? I'll, I'll share my screen. We'll look at my, uh, my Twitter analytics if you wanna see. **Host** [1:25:52] It's on your Substack. Yeah, yeah. **Steve Ruiz** [1:25:53] I'll... Okay. Oh, yeah, it is, it is. I need to update that image. I've done it once already. Uh, yeah, if you, if you're a, if you're a Substack reader, the tldraw Substack, 72 hours into this big Make Real virality explosion, I, I sat down and wrote a blog post, and I, uh, I wanted to- **Host** [1:26:11] Yeah **Steve Ruiz** [1:26:11] ... at least capture that, that vibe, um, of what it felt like in the middle of that, that hurricane. Uh- **Host** [1:26:17] Yeah **Steve Ruiz** [1:26:17] ... so it's, it's a pretty fun one. It's good to read back. **Host** [1:26:21] Uh, well, I'm sure it's not the last time we'll see you do something crazy viral. Uh, I'm sure that a lot of people will be exploring tldraw. I hope a lot of people... Honestly, um, one thing I'm thinking about is, like, em- embedding tldraw into my, my input box. Like, why can't, why can't tldraw be, like, you know, part of the, the, the input? **Steve Ruiz** [1:26:40] Hey, I'm, I'm talking to, uh, the, the good folks over at OpenAI tomorrow. Fingers crossed- **Host** [1:26:45] Ooh **Steve Ruiz** [1:26:45] ... maybe we, uh, maybe we get it in, inside of, uh, ChatGPT or something. 'Cause yeah, like, I don't know, I wanna- **Host** [1:26:52] Okay. I need a 20 more factor. **Steve Ruiz** [1:26:54] I don't, can it? Yeah. **Host** [1:26:56] Uh, yeah. **Steve Ruiz** [1:26:57] We, you c- like, what? You wanna, like, take a drawing or take a photo and then annotate it? Or like, you know, sketch something out. **Host** [1:27:03] Just like instead of, instead of text input. **Steve Ruiz** [1:27:03] You should be able to do that. **Host** [1:27:05] Yeah, exactly. **Steve Ruiz** [1:27:06] Yeah, it's just a good, it's just a good thing. Uh, yeah. The, the pe- people cry out for it. I, uh, you know, can't build it fast enough. **Host** [1:27:16] Yeah. Um, well, thank you for, uh, inspiring the rest of us. Uh, thank you for everything, and I'm sure we'll, we'll hear from, more from you over, over the next few years. Uh, so, uh, thanks. Thanks for your time. **Steve Ruiz** [1:27:26] Awesome. Thank you for your time. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.