# AI Engineering for Art - with comfyanonymous Latent Space · 2025-01-04 Comfy Anonymous, the anonymous creator of ComfyUI, details in his first-ever podcast interview how his node-based image generation tool overtook Automatic1111 through superior memory management and early support for SDXL, becoming the de facto interface for advanced diffusion workflows. He started coding on January 1, 2023, and released the first version on January 16, driven by a desire to chain models and experiment with custom samplers. His work at Stability AI from June 2023 ensured ComfyUI efficiently handled SDXL, which leaked via early access and forced users from less powerful GPUs to adopt his tool. He discusses model preferences: Flux for consistency, SD 3.5 for creativity, and SD1.5 remaining popular. For video, he highlights Mochi as a 'true' video model with 3D latents, already implemented in Comfy. Now with a core team, ComfyUI is targeting a v1 release with an easy installer on Windows and Mac, while planning monetization through cloud and enterprise features—but keeping the open-source core free. ## Questions this episode answers ### How did ComfyUI overtake Automatic1111 to become the standard for Stable Diffusion? Comfyanonymous explains that when Stability AI released SDXL, Automatic1111 was so inefficient that most users couldn't run it on their GPUs. ComfyUI, with better memory management and support for model chaining, was the only tool that could handle SDXL on lower-end hardware, forcing the community to adopt it. This pivotal moment, combined with the SDXL 1.0 rollout, cemented ComfyUI's dominance. [47:42](https://addtry.com/8364245d-9333-4df6-ae3e-0a2cf5e4c18d?t=2862000) ### Why is ComfyUI called 'Comfy' and who created it? The creator, an anonymous former Stability AI engineer known as comfyanonymous, began building ComfyUI in January 2023 to experiment with chaining diffusion models for high-resolution images, after finding Automatic1111's code inflexible. The name 'Comfy' originated from his online persona, which people associated with the cozy, 'comfy' style of his generated images. He released the first version on GitHub on January 16, 2023. [4:41](https://addtry.com/8364245d-9333-4df6-ae3e-0a2cf5e4c18d?t=281000) ### What makes ComfyUI's node-based workflow more powerful than traditional interfaces like Automatic1111? Comfyanonymous designed ComfyUI with a node graph interface that maps the diffusion process visually, enabling tasks like chaining different models, area conditioning, and custom pipeline orchestration. Its backend includes an asynchronous queue, smart GPU memory management that minimizes loading delays, and a modular architecture that separates frontend from backend, making it highly extensible and efficient for complex AI art workflows. [26:22](https://addtry.com/8364245d-9333-4df6-ae3e-0a2cf5e4c18d?t=1582000) ## Key moments - **[0:00] Beginnings** - [1:11] Comfy Anonymous discovered Stable Diffusion in October 2022 and started using Automatic1111's Web UI. - [2:37] A node graph interface is the natural way to represent the diffusion process. - [4:21] Comfy built his own interface because Auto's codebase couldn't chain different models for the high-res fix. - [5:03] ComfyUI was first released on GitHub on January 16, 2023. - [6:05] Area conditioning—applying prompts to different image regions—was one of ComfyUI's first popular features. - [7:11] A month later, the Multi-Diffusion paper described the same area composition technique Comfy had already implemented. - **[10:06] Model Wars** - [10:17] Flux is the best image generation model; Stable Diffusion 3.5 is more creative for tasks requiring variety. - [12:40] Stable Cascade was a good model but was overshadowed by the Stable Diffusion 3 announcement one week later. - [14:14] The community only adopts a new model when there is a significant improvement, not for incremental gains. - [15:32] Comfy users evaluate models by vibes: 'I think this image is nice.' - **[16:17] LoRAs & Prompts** - [17:32] Textual inversion teaches the model a new word vector for a concept, but adoption has faded. - [20:29] LongCLIP, which accepts 256-token prompts, is supported natively in ComfyUI. - [21:45] Prompt weighting works on CLIP-based text encoders but fails on larger ones like T5-XXL. - [23:47] LoRAs are weight differences represented by low-rank matrices, making fine-tuning efficient and portable. - **[24:39] Design Philosophy** - [26:08] Comfy chose to build a powerful, not necessarily easy-to-use interface while everyone else targeted simplicity. - **[27:05] Engine Room** - [27:12] The biggest engineering challenge in ComfyUI is GPU memory management to avoid slow NVIDIA driver paging on Windows. - [33:57] Most ComfyUI users have NVIDIA GPUs; AMD support on Windows is poor due to driver and PyTorch ROCm issues. - **[35:08] Node Playground** - [35:41] ComfyUI exposes sampler nodes from basic to advanced, letting users control every diffusion parameter. - [37:49] Key sampling parameters: lower steps for speed, and CFG adjusts image contrast. - [39:41] A user built a playable Wolfenstein 3D inside ComfyUI with AI-generated textures piped through the node graph. - **[41:35] Video Models** - [42:05] Mochi is a true video diffusion model with 3D latents and temporal VAE, unlike Stable Video Diffusion's 2D latents. - [44:19] Stable Diffusion 3.5, Mochi, and Omnigen all released on the same day, causing them to drown each other out. - **[44:31] Comfy's Arc** - [45:54] SDXL's staggered release—code first, then leaked checkpoints—made ComfyUI the only way to run it on regular GPUs, driving adoption. - [48:48] ComfyUI V1 will package the new frontend with one-click installs for Windows and Mac while keeping the core open source. - [50:36] Comfy welcomes other startups building on ComfyUI, saying it strengthens the ecosystem. - **[50:58] What's Next** ## Speakers - **Swyx** (host) - **Alessio** (guest) - **Comfy Anonymous** (guest) ## Topics Image Generation, Open Source Tools ## Mentioned AMD (company), NVIDIA (company), Stability AI (company), AnimateDiff (product), Automatic (product), CogVideo (product), ComfyUI (product), Flux (product), Gradio (product), IPAdapter (product), Krita (product), Light Graph (product), LongCLIP (product), Mochi (product), PyTorch (product), SDXL (product), Stable Cascade (product), Stable Diffusion (product) ## Transcript ### Beginnings **Alessio** [0:04] Hey everyone, welcome to the Latent Space Podcast. This is Alessio, partner and CTO of Decibel Partners, and I'm joined by my co-host, swyx, founder of Smol AI. **Swyx** [0:13] Hey everyone. We are in the Chroma studio again, but with our first ever anonymous guest. Comfy Anonymous, welcome. **Comfy Anonymous** [0:20] Yeah. Well, hello. **Swyx** [0:21] I feel like that's your full name. You, you just go by Comfy, right? **Comfy Anonymous** [0:24] Yeah, well, a lot of people just call me Comfy even though even if- even when they know my real name. They say, "Hey, hey Comfy." **Alessio** [0:32] Yeah. swyx is the same. Like, you know, not a lot of people call you Sean. **Swyx** [0:36] Yeah, it's-- Yeah, you have a professional name, right- **Alessio** [0:38] Yeah **Swyx** [0:38] ... that people know you by, and then, then you have a legal name. Yeah, it's fine. How do I phrase this? Like I think a lot of people who are in the know know that Comfy is like the tool for image generation and now other multimodality stuff. I would say that when I first got started with Stable Diffusion, the star of the show was Automatic 11 11, right? And I, I actually looked back at my notes from 2022-ish, like Comfy was already getting started back then, but it was kind of like the up-and-comer and like your main feature was a flowchart. Can you just kind of rewind to that moment, so that, that year and like, you know, how, how you looked at the landscape there and decided to start Comfy? **Comfy Anonymous** [1:11] Yeah. I discovered Stable Diffusion in, uh, 2022, in October 2022, and, uh, well, I kind of started playing around with it. Yes, I-- And back then I was using Automatic, which was what everyone was using back then, and I-- So I started with that. 'Cause I, I had the-- 'Cause when I started, I had no idea like how diffusion models work, how any of this works. So **Swyx** [1:37] Oh, yeah. What was your prior background as an engineer? **Comfy Anonymous** [1:40] Uh, just a software engineer. Yeah, boring software engineer. **Swyx** [1:44] But like any, any image stuff, any orchestration- **Comfy Anonymous** [1:48] No **Swyx** [1:48] ... distributed systems, GPUs? **Comfy Anonymous** [1:50] No, I was doing, uh, basically nothing interesting. **Swyx** [1:55] Crud web development. **Comfy Anonymous** [1:57] Yeah. Well, not web development. Just, yeah, some basic-- maybe some basic like automation stuff and, uh- **Swyx** [2:03] Okay **Comfy Anonymous** [2:04] ... just, yeah. But, uh, no, like, uh, no big companies or anything. **Swyx** [2:09] Yeah, but like already some interest in automations, probably a lot of Python. **Comfy Anonymous** [2:13] Yeah, yeah, of course, Python. But, uh, like I wasn't actually used to like, uh, the Node graph interface- **Swyx** [2:20] Mm **Comfy Anonymous** [2:21] ... before I started ComfyUI. It was just, uh, I just thought it was like, "Oh, like what's the best way to represent the diffusion process in the user interface?" And then like, oh, well, like naturally, this is the best way I found. And this was, uh, like with the Node interface. **Swyx** [2:41] Mm-hmm. **Comfy Anonymous** [2:41] So how I got started was, uh, yeah, so basic October 2022, just, uh, like I hadn't written a line of PyTorch before that. **Swyx** [2:52] Mm-hmm. **Comfy Anonymous** [2:52] So it's, uh, completely new. What happened was I kind of got addicted to generating images. **Alessio** [2:59] As we all did. **Comfy Anonymous** [3:00] Yeah. And then I started experimenting with, uh, like the high-res fix in Auto, which was, uh, for those that don't know, the high-res fix is just to gen-- Since the diffusion models back then could only generate at low resolution. **Swyx** [3:17] Mm-hmm. **Comfy Anonymous** [3:17] So what you would do, you would generate low-resolution image, then upscale, then kind of pa- refine it again, and that was kind of the hack to generate high-resolution images. I really liked generating like higher resolution images, so I was experimenting with that. And so I modified the code a bit. Okay, what happens if I, if I use different samplers on the second pass? **Swyx** [3:47] Mm-hmm. **Comfy Anonymous** [3:47] So I must edited the code of Auto. So what happens if I use a different sampler? What happens if I use a different, uh, like a different settings and a different number of steps? Uh, and because back then the high-res fix was very basic. **Swyx** [4:04] Mm-hmm. **Comfy Anonymous** [4:04] Just, uh, so. **Swyx** [4:06] Yeah. Now there's a whole library of just, uh, the upsamplers that's very popular. **Comfy Anonymous** [4:09] Yeah, yeah, right. I think, yeah, I think they added a bunch of, uh, of, uh, options to the high-res fix since, uh, since, since then. But before that it was just so basic, so I wanted to go further. I wanted to try, okay, what happens if I use a different model for the second, the second pass? And then, well, then the Auto code base was, wasn't good enough for it. Like it would have been, uh, harder to implement that in the Auto interface than to create my own interface. So that's when I decided to create my own. **Alessio** [4:45] And you were doing that mostly on your own when you started, or did you already have kind of like a subgroup of people? **Comfy Anonymous** [4:49] No, I was, uh, on my own. 'Cause it was just me experimenting with stuff. So yeah, that was, uh, bas-- So I started writing the code January 1, 2023. **Alessio** [5:03] Mm-hmm. **Comfy Anonymous** [5:03] And then I released the first version on GitHub January 16, 2023. That's how things got started. **Alessio** [5:12] And was, was the name ComfyUI right away or- **Comfy Anonymous** [5:14] Yeah, yeah. ComfyUI. The reason the name, my name is Comfy is people thought my pictures were comfy. So I just, uh, just name it, uh, uh, it's my ComfyUI. So yeah, that's, uh- **Swyx** [5:28] Is there a particular segment of the community that you targeted as users, like more intensive workflow artists, you know, compared to the Automatic crowd or, you know? **Comfy Anonymous** [5:38] This was my way of like experimenting with, uh, with new things, like the high-res fix thing I mentioned, which was like in Comfy, the first thing you could easily do was just chain different models together. And then one of the first things-- I think the first times it got a bit of popularity was when I started experimenting with the different, like applying prompts to different areas of the image. **Swyx** [6:04] Mm-hmm. **Comfy Anonymous** [6:05] Yeah, I called it Area Conditioning. Posted it, it on Reddit, and it got a, a bunch of upvotes. So I think that's when, like, when people first, uh, learned of ComfyUI. **Alessio** [6:17] Mm. **Swyx** [6:18] Is that mostly, like, fixing hands? **Comfy Anonymous** [6:20] Oh, no, that was just, uh, like, let's say... Well, it was very-- Well, it still is kind of difficult to, like... Let's say you want a, a mountain. You have an image, and then you, "Okay, I want a mountain here, and I want, uh, like, a, a fox here." **Swyx** [6:38] Mm. **Comfy Anonymous** [6:38] Like- **Swyx** [6:38] Yeah, so compositing the image. **Comfy Anonymous** [6:40] Yeah. By the way, it was very easy. It was just like, oh, when you run the diffusion process, you kind of generate, okay, you do pass, one pass through the diffusion model. Every step you do one pass. Okay, this place of the image with this prompt, this place, place of the image with the other prompt, and then the entire image with another prompt, and then just average everything together every step, and that was, uh, area composition, which I call it. And then, then a, a month later, there was a paper that came out called Multi-Diffusion, which was the same thing. But, uh, yeah. That's, uh- **Alessio** [7:21] Could you do area composition with different models or, or because you're averaging out, you kinda need the same model? **Comfy Anonymous** [7:27] Well, you could do it with-- But yeah, I hadn't implemented it for different models. But, uh, you, you can do it with, uh, with different models if you want, as long as the models share the same latent space. **Alessio** [7:40] Mm-hmm. **Comfy Anonymous** [7:41] Like you- **Swyx** [7:42] We h- we're supposed to ring a bell every time someone says latent space. **Alessio** [7:45] Yeah, it's from- **Comfy Anonymous** [7:46] Yeah. Like, for example, you couldn't use, like, XL and SD 1.5 'cause those have a different latent space. But, like, uh, yeah, like SD 1.5 models, different ones, you could, you could do that, uh- **Swyx** [8:00] There's some models that try to work in pixel space, right? **Comfy Anonymous** [8:03] Yeah. They're very slow. **Swyx** [8:06] Of course. **Comfy Anonymous** [8:06] That's the problem. That, that's the, the reason why Stable Diffusion actually became, like, popular, like, 'cause-- was because of the latent space. **Swyx** [8:15] Yeah, small in... Yeah. **Comfy Anonymous** [8:16] Yeah. **Swyx** [8:17] Because it used to be latent diffusion models, and then they trained it up. **Comfy Anonymous** [8:20] Yeah, 'cause the pixel, pixel diffusion models are just, uh, too slow, so. **Swyx** [8:25] Yeah. Have you ever tried to talk to, like, uh, like Stability, the latent diffusion guys, like, you know, Robin Rombach, that, that crew? **Comfy Anonymous** [8:32] The-- Yeah. Well, I used to work at Stability. **Swyx** [8:35] Oh, I actually didn't know. **Comfy Anonymous** [8:36] Yeah. I used to work at Stability. I got, uh, I got hired, uh, in June 2023. **Swyx** [8:43] Ah. That's the part of the story I didn't know about. Okay. **Comfy Anonymous** [8:46] Yeah. So the, the reason I was hired is because they were doing, uh, SD XL at the time. **Swyx** [8:51] Mm. **Comfy Anonymous** [8:52] And they were-- Basically, SD XL, I don't know if you remember, it was a base model and then a refiner model. **Swyx** [8:58] Mm-hmm. **Comfy Anonymous** [8:59] Basically, they wanted to experiment, like, chaining them together, and then, uh, they saw, "Oh." **Swyx** [9:05] Right. **Comfy Anonymous** [9:06] "Can-- Oh, this, we can use this to do that. Well, let's hire that guy." **Swyx** [9:11] But they didn't, they didn't pursue it for, like, SD3. **Comfy Anonymous** [9:13] What do you mean? **Swyx** [9:14] Like, the SD XL approach. **Comfy Anonymous** [9:16] Yeah. The reason for, for that approach was because basically they had two models, and then they wanted to publish both of them. So they, they trained one on lower time steps, which was the refiner model, and then they-- the first one, well, was trained normally. And then they w-- During their test, they realized, "Oh, like, if we string these models together, our, like, quality increases. So let's publish that." **Swyx** [9:47] It worked. **Comfy Anonymous** [9:47] Yeah. But, uh, like, right now, I don't think many people actually use the refiner anymore, even though it is actually a full diffusion model. Like, you can use it on its own, and it's gonna generate images. I don't think anyone-- People have, uh, mostly forgotten about it. But, uh- **Alessio** [10:06] Can we talk about models a little bit? So Stable Diffusion obviously is the most known. ### Model Wars **Comfy Anonymous** [10:09] Yeah. **Alessio** [10:09] I know Flux has gotten a lot of- **Comfy Anonymous** [10:12] Yeah **Alessio** [10:12] ... traction. Are there any underrated models that people should use more, or what's the state of the union? **Comfy Anonymous** [10:17] Well, the, the latest, uh, state-of-the-art, at least, yeah, for images, there's, uh, yeah, there's Flux. Uh, there's also SD 3.5. SD 3.5 is two models. There's a, there's a small one, 2.5B, and there's the bigger one, 8B. So it's, it's smaller than Flux, so... And it's more, uh, creative in a way. **Alessio** [10:42] Mm-hmm. **Comfy Anonymous** [10:43] But Flux, yeah, Flux is the best. People should give SD 3.5 a try 'cause it's, uh, it's different. I won't say it's better. Well, it's better for some, like, specific use cases. Like, if you want some-- to make something more, like, creative, maybe SD 3.5. If you want to make something more consistent, then Flux is probably better. **Swyx** [11:07] Do you ever consider supporting the closed source model APIs? **Comfy Anonymous** [11:11] Uh, well, they-- we do support them with the custom nodes. We actually have some, uh, official custom nodes from, uh, different- **Swyx** [11:20] Ideogram. **Comfy Anonymous** [11:21] Yeah. Have- **Swyx** [11:22] I guess DALL-E would have one. **Comfy Anonymous** [11:24] Yeah. That's, uh, it's just not I'm not the person that handles that. **Swyx** [11:28] Mm-hmm. Sure, sure. Quick question on, on SD. There's a lot of community discussion about the transition from SD 1.5 to SD 2, and then SD 2 to SD 3. People still, like, you know, very loyal to the previous generations of SDs? **Comfy Anonymous** [11:42] Uh, yeah. SD 1.5 then still has a lot of, uh- ... a lot of users. **Swyx** [11:47] The last based model. **Alessio** [11:50] Yeah. **Comfy Anonymous** [11:50] Yeah. Then SD 2 was mostly ignored as it wasn't, uh, it wasn't a big enough improvement over the previous one. **Swyx** [11:59] Okay, so SD 1.5, SD3, Flux, and whatever else **Comfy Anonymous** [12:03] Yeah, SDXL **Swyx** [12:04] SDXL **Comfy Anonymous** [12:04] SDXL, that's the main one **Swyx** [12:05] Stable Cascade, you know? **Comfy Anonymous** [12:06] Stable Cascade, that was a good model, but, uh, that's-- uh, the, the problem with that one is, uh, it got, uh, like SD3 was announced one week after **Swyx** [12:16] Yeah, it was like a weird release, uh... What was it like inside of Stability, actually? **Comfy Anonymous** [12:21] I don't know. **Swyx** [12:22] I mean, statute of limitations expired, you know, management has, has moved somewhere. It's easier to talk about now. **Comfy Anonymous** [12:27] Yeah. I think the inside Stability, actually, that model was ready, uh, like three months before, but it got, uh, stuck in, uh, red teaming. So basically, the problem that if that model had released or was supposed to be released- ...by the authors, then it would probably have gotten very popular since it's a, it's a step up from SDXL. But it got all of its momentum stolen by the SD3 announcement, so people kind of- **Swyx** [12:56] Yeah **Comfy Anonymous** [12:57] ... didn't develop anything on top of it, even though it's, uh, yeah. It was a good model, at least, uh, completely mostly ignored for some reason. Like- **Swyx** [13:08] It seem-- I think the naming as well matters. It seemed like a branch off of the main ra- main- **Comfy Anonymous** [13:14] Yeah **Swyx** [13:15] ... tree of development. **Comfy Anonymous** [13:16] Yeah. Well, it was different researchers- **Swyx** [13:18] Yeah **Comfy Anonymous** [13:18] ... that did it. Like- **Swyx** [13:19] Different branch **Comfy Anonymous** [13:19] ... yeah. **Swyx** [13:20] Yeah. **Comfy Anonymous** [13:20] V- very, like a good model. Like, it's the Wurster-Schun authors. I'm not sure if I'm pronouncing it correct. **Swyx** [13:27] Wurschin, yeah. **Comfy Anonymous** [13:27] Wurschin, yeah. Yeah, they- **Swyx** [13:29] I actually met them in, uh, Vienna. **Comfy Anonymous** [13:31] Yeah, they worked at Stability for a bit, and they left right after the Cascade release. **Swyx** [13:36] This is Dustin, right? **Comfy Anonymous** [13:37] No. **Swyx** [13:38] Uh, Dustin's SD3. Yeah. **Comfy Anonymous** [13:39] No, uh, Dustin is, uh, SD3, SDXL. That's, uh, Pablo and- **Swyx** [13:44] Yeah. Yeah, yeah **Comfy Anonymous** [13:45] ... uh, Dome. Besides, I think I'm pronouncing his name correctly. **Swyx** [13:49] Ah. **Comfy Anonymous** [13:49] Yeah, that's, uh- **Swyx** [13:50] Yeah, very cool **Comfy Anonymous** [13:50] ... you know, very, very good. **Swyx** [13:52] It seems like the community is very-- they move very quickly. **Comfy Anonymous** [13:55] Yeah. **Swyx** [13:56] Like, when there's a new model out, they just drop whatever the current one is, and they just all move wholesale over. Like, they don't really stay to explore the full capabilities. **Comfy Anonymous** [14:04] Mm. **Swyx** [14:04] Like if, if the Stable Cascade was that good, they would have A/B tested a bit more. Instead, they're like, "Okay, SD3 is out. Make this go." You know? **Comfy Anonymous** [14:12] Well, I find the opposite actually. **Swyx** [14:14] Okay. **Comfy Anonymous** [14:14] The community doesn't-- like they only jump on a new model when there's a significant improvement. **Swyx** [14:19] I see. **Comfy Anonymous** [14:20] Like, if there's, uh, only like a incremental improvement, which is what, uh, most of these models are, are going to have, especially if you-- 'cause, uh, stay the same parameter count, uh- **Swyx** [14:33] Yeah **Comfy Anonymous** [14:33] ... like you're not gonna get a massive improvement, uh, into like... Unless there's something big that, that changes, so, uh, yeah. **Swyx** [14:42] And how are they evaluating these improvements? Like, um, because this-- it's a, it's a whole chain of, you know, Comfy workflows. **Comfy Anonymous** [14:48] Yeah. **Swyx** [14:49] How does, how does one part of the chain actually affect the whole process? **Comfy Anonymous** [14:53] Are you talking on the model side specific? **Swyx** [14:55] Model specific, right. But like once you have your own whole workflow based on a model, it's very hard to move. **Comfy Anonymous** [15:02] Uh, not... Well- **Swyx** [15:04] No? **Comfy Anonymous** [15:04] ... not really. **Swyx** [15:05] Maybe not. **Comfy Anonymous** [15:05] Depends on your, uh, depends on the specific kind of the workflow. **Swyx** [15:09] Yeah. So I do a lot of like text and image. **Comfy Anonymous** [15:12] Yeah. When you do change, uh, like most workflows are kind of gonna be compatible between different models. It's just like you might have to completely change your prompt, completely change. **Swyx** [15:24] Okay. Well, I mean, then maybe the question is really about evals. Like what does, uh, the Comfy community do for evals? Just, you know- **Comfy Anonymous** [15:32] Well, they, they don't really do. They-- It's more like, oh- **Swyx** [15:36] Vibe check **Comfy Anonymous** [15:36] ... I think this image is nice. **Swyx** [15:38] Yeah. **Comfy Anonymous** [15:38] So that's, uh **Swyx** [15:39] They just subscribe to FOOFER AI and just see like, you know, what FOOFER is doing. **Comfy Anonymous** [15:44] Yeah. They just, they, they just generate like, like I don't see anyone really doing like, uh, at least on the Comfy side, Comfy users, they-- it's more like, oh, generate images and see, oh, this one's nice. **Swyx** [15:56] Yeah, yeah. **Comfy Anonymous** [15:56] This is nice. **Swyx** [15:57] Yeah, vibes. **Comfy Anonymous** [15:58] Yeah. Yeah. It's not, uh, like the, the more, uh, like, uh, scientific, uh, like, uh, like checking, that's more on specifically on like model side if, uh... Yeah. But there is a lot of, uh, vibes also 'cause it is a, like- ... a artistic, uh... You can create a very good model that doesn't generate nice images. 'Cause mo- most images on the internet are ugly. So if you- ### LoRAs & Prompts **Swyx** [16:29] Mm **Comfy Anonymous** [16:29] ... if that's like, if you just, "Oh, I, I have the best model that can, like, it's super smart. I trained it on all the, like... I've trained it on just all the images on the internet." The images are not gonna look good. So- **Swyx** [16:43] Yeah. Yeah. **Comfy Anonymous** [16:44] They're gonna be very consistent. But yeah, peop- like it's not gonna be like the, the look that people are gonna be expecting from, uh, from a model, so yeah. **Swyx** [16:55] Can we talk about LoRAs? 'Cause we talked, we talked about models, then like the next step is probably LoRAs. Before-- Uh, actually, I'm kinda curious how LoRAs entered the tool set of the image community, because the LoRA paper was 2021, and then like there was like other methods like textual inversion that was popular at, at the early SD stage. **Comfy Anonymous** [17:14] Yeah, I can explain the difference between that. Like textual inversions, that's, uh, basically what you're doing is you're, you're training a... 'cause, well, yeah, Stable Diffusion, you have the diffusion model, you have the text encoder. So basically, what you're doing is training a vector that you're gonna pass to the text decoder. It's basically you're training a new word. **Swyx** [17:37] Yeah. It's a little bit like representation engineering now. **Comfy Anonymous** [17:40] Yeah. Basically, yeah, you're just, uh... So yeah, if you know how, uh, like, uh, the text encoder works, basically you have, uh, you, you take your, your words of your prompt, you convert those into tokens with the tokenizer, and those are converted into vectors. Basically, yeah, each token represents a different vector, so each word represents a vector, and those Depending on your words, that's the list of vectors that get passed to the text encoder, which is just, uh, yeah, just a stack of, uh, of attention. Like, basically, it's, uh, very close to LLM architecture. Yeah. Yeah, so basically what you're doing is just training a new vector. We're saying, "Well, I have all these images, and I want to know which word does that represent." And it's gonna get-- Like, you train this vector, and then, and then when you use this vector, it hopefully generates, uh, like something similar to your images. Yeah. **Swyx** [18:44] Yeah. I would say it's, like, surprisingly sample efficient in picking up the concept that you're trying to train it on. **Comfy Anonymous** [18:49] Yeah. Well, people have kind of stopped doing that. **Swyx** [18:52] Yeah. **Comfy Anonymous** [18:52] Even though, uh, back at, like, when I was at Stability, we, we actually did train internally some, like, textual versions on, like, T5-XXL. **Swyx** [19:02] Mm. **Comfy Anonymous** [19:02] Actually worked pretty well. But, uh, for some reason, yeah, people don't use them. And also, they might also work, uh, like, like... Yeah, that's just something you'd probably have to test, but maybe if you train a textual version, like on T5-XXL, it might also work with all the other models that use T5-XXL. 'Cause same thing with, uh, like, uh, like the textual versions that, uh, that were trained for SD 1.5, they also kind of work on SDXL, because SDXL has the, has two text encoders, and one of them is the same as the, as the SD 1.5 Clip L. So those, they actually w- they don't work as strongly 'cause they're only applied to one of the text encoders. But, uh, and the same thing for SD3. Three-- SD3 has three text encoders, so- **Swyx** [19:53] Mm. **Comfy Anonymous** [19:53] It works. It's still w-- You can still use your text conversion SD 1.5 on SD3, but it's just a lot weaker because now there's three text encoders, so it's gets even more diluted. Yeah. **Swyx** [20:06] Yeah. Do people experiment a lot on, on... Just on the Clip side, uh, there's like SigLIP, there's BLIP. Like, do people experiment a lot on, on those? **Comfy Anonymous** [20:13] Uh, well, you, you can't really replace- **Swyx** [20:14] Yeah, 'cause they're, they're trained together, right? **Comfy Anonymous** [20:16] Yeah, they're trained together, so you, you can't, uh... Like, well, what I've seen people experimenting with is, uh, LongCLIP. So basically someone, uh, fine-tuned the CLIP model to accept longer prompts. **Swyx** [20:28] Oh. **Comfy Anonymous** [20:29] Yeah. **Swyx** [20:29] It's kind of like long context fine-tuning. **Comfy Anonymous** [20:31] Yeah. So, so like it's, it's actually supported in core Comfy. **Swyx** [20:36] How long is long? **Comfy Anonymous** [20:37] Regular CLIP is 77 tokens. **Swyx** [20:40] Yeah. **Comfy Anonymous** [20:40] LongCLIP is 256. **Swyx** [20:43] Okay. **Comfy Anonymous** [20:44] So, but the hack that, uh, like you've-- if you've used Stable Diffusion 1.5, you've probably noticed, "Oh, it still works if I, if I use long prompts," prompts longer than 77 words. Well, that's because the hack is to just, uh, well, you split, you split it up in chunks of seven. Your whole- **Swyx** [21:05] Mm. **Comfy Anonymous** [21:05] Your big prompt. Let's say you, you give it like the massive text, like the Bible or something. **Swyx** [21:12] Oh, my. **Comfy Anonymous** [21:12] And it would split it up in chunks of 77- **Swyx** [21:16] Yeah. It's like- **Comfy Anonymous** [21:16] And then just pass each one through the, the CLIP. **Swyx** [21:19] Yeah. **Comfy Anonymous** [21:19] And then just concatenate- **Swyx** [21:21] Revive **Comfy Anonymous** [21:22] ... everything together at the end. It's not- **Swyx** [21:24] Ooh **Comfy Anonymous** [21:24] ... ideal, but it actually- **Swyx** [21:25] That's messy **Comfy Anonymous** [21:26] ... works. **Swyx** [21:27] Like the positioning of the words really, really matters then, right? **Comfy Anonymous** [21:30] Yeah. **Swyx** [21:30] Like, this is why order matters in prompts. **Comfy Anonymous** [21:33] Yeah. Yeah, like it, it works, but it's, it's not ideal, but it's what people expect. Like if, if someone gives- **Swyx** [21:41] Yeah **Comfy Anonymous** [21:42] ... a huge prompt, they expect at least some of the concepts at the end- **Swyx** [21:46] Yeah **Comfy Anonymous** [21:46] ... to be like present in the image. But usually when they give long prompts, they, they don't, they like, they don't expect, uh, like detail, I think. So that's why it works very well. **Swyx** [21:59] And while we're on this topic, uh, prompt weighting, neg- negative prompting, all, all sort of similar part of this- **Comfy Anonymous** [22:05] Yeah **Swyx** [22:05] ... layer of the stack. **Comfy Anonymous** [22:06] Yeah. The, the hack for that which works on CLIP, like it w- basically it's just, uh, for SD 1 point f-- well, for SD 1.5, the prompt weighting works well because, uh, Clip L is a, is not a very deep model. So you have a very high correlation between you have the input token, the index of the input token vector, and the output token. They're very-- the concepts are very close, closely linked. So that means if you interpolate the vector from what... Well, the, the way ComfyUI does it is it has, okay, you have the vector. You have a empty prompt. So you have a, a child, like a CLIP output for the empty prompt, and then you have the one for your prompt, and then it interpolates from that depending on your prompt weight, the weight of your, of your tokens. So, so if you... Yeah. So that's how it, how it does, uh, prompt weighting, but this stops working the deeper your text encoder is. **Swyx** [23:16] Mm. **Comfy Anonymous** [23:17] So on T5-XXL, it doesn't work at all, so. **Swyx** [23:20] Wow. Is that a problem for people? I mean, 'cause I, I'm used to just move, moving up numbers. **Comfy Anonymous** [23:25] Probably not- **Swyx** [23:25] Yeah **Comfy Anonymous** [23:25] ... 'cause, uh, well- **Swyx** [23:27] So you just use words to describe, right? 'Cause it's a bigger language model. **Comfy Anonymous** [23:30] Yeah. Yeah. So honestly, it might be good, but I haven't seen many complaints on Flex that, uh, it's not working. So- **Swyx** [23:40] Yeah **Comfy Anonymous** [23:40] ... 'cause I guess people can sort of, uh, get around it with, uh, with language. So, uh, yeah. **Swyx** [23:47] Yeah. And then coming back to LoRAs, now the, the popular way to, to customize models is LoRAs. And I s- I saw you also support LoCon and LoHa, which I've never heard of before. **Comfy Anonymous** [23:56] There's a bunch of, uh... 'Cause, uh, what, what the LoRA is essentially is, uh, instead of, uh, like, okay, you have your, your model, and then you want to fine-tune it. So instead of, uh, like what you could do is you could fine-tune the entire thing. But that's- **Swyx** [24:11] Yeah, full fine-tune. Yeah **Comfy Anonymous** [24:12] ... but that's a bit, uh- heavy. So to speed things up and make things less heavy, what you can do is just fine-tune some smaller weights. Like basically two, two matrices that when you multiply, like two low-rank matrices, that when you multiply them together gives a-- represents a difference between trained weights and your base weights. **Alessio** [24:39] Mm-hmm. ### Design Philosophy **Comfy Anonymous** [24:39] So by training those two smaller matrices, that's a lot less heavy. **Alessio** [24:46] Yeah. And they're portable, so you can share them. **Comfy Anonymous** [24:48] Yeah. **Alessio** [24:48] It's like easier. **Comfy Anonymous** [24:49] And also smaller. Yeah. That's the-- how LoRAs work. So basic-- So when, when inferencing, you can inference with them pretty efficiently, like how ComfyUI does it. It just-- When you use a LoRA, it just applies it straight on the weights so that there's only a small delay at the be- like before the sampling to-- when it applies the weights and then it just same speed as, uh, as before. So for, for inference, it's, it's not that bad. But, uh... And then you have, uh... So basically, all the LoRA types like LoHA, LoVA, everything, that's just different ways of, uh, representing that, uh. Like, basically, you can call it kind of like compression even though it's not really compression. It's just different ways of represent, like just, okay, I want to train a different on the-- difference on the weights. What's the best way to represent that difference? There's the basic LoRA, which is just, "Oh, let's multiply these two matrices together," and then there's all the other ones, which are all different, uh, algorithms. So yeah. **Alessio** [25:58] Yep. So let's talk about what ComfyUI actually is. I think most people have heard of it. Some people might have seen screenshots. I, I think fewer people have built very complex workflows. **Comfy Anonymous** [26:08] Yeah. **Alessio** [26:08] So when you started, Automatic was like the super simple way. What were some of the choices that you made? So the node workflow, is there anything else that stands out as like this was like a unique take on how to do image generation workflows? **Comfy Anonymous** [26:22] Well, I feel like, yeah, back then everyone was trying to make like easy-to-use interface. Then I'm like, "Well, everyone's trying to make an easy-to-use interface." **Swyx** [26:33] Let's make a hard-to-use interface. **Comfy Anonymous** [26:37] Like so, like, I like, I don't need to do that. **Alessio** [26:41] Yeah. **Comfy Anonymous** [26:43] Everyone else doing it, so let me try something, uh, like let me try to make a powerful interface- **Swyx** [26:49] Yeah, yeah **Comfy Anonymous** [26:49] ... that's, uh, not easy to use, so. **Swyx** [26:53] So like, yeah, there's a sort of node execution engine. Your README actually lists, uh, this really good list of features of things you prioritize, right? Like, um, let me see, like, uh, sort of re-executing from a, from any parts of- ### Engine Room **Comfy Anonymous** [27:05] Yeah **Swyx** [27:05] ... this workflow that was changed, asynchronous queue system, smart memory management. Like, all, all this seems like a lot of engineering that- **Comfy Anonymous** [27:12] Yeah, there's a lot of engineering in the, in the back end to make, uh, things, uh... 'Cause I was always, uh, focused on making things work locally very well 'cause that's-- 'cause I was using it, uh, locally, so everything, uh... So there's a lot of, uh, a lot of thought and work in, like, getting everything to run as well as possible. So yeah, ComfyUI is actually more of a back end- **Swyx** [27:39] Mm-hmm **Comfy Anonymous** [27:39] ... or at least, uh, well, now, now the front end's getting a, a lot more development . But, but before, before it was-- I was pretty much only focused on the back end. **Swyx** [27:50] Yeah. So v0.1 was only August this year- **Comfy Anonymous** [27:54] Yeah, before there was- **Swyx** [27:55] With the new front end **Comfy Anonymous** [27:56] ... no versioning, so. **Swyx** [27:57] Yeah, yeah. **Comfy Anonymous** [27:57] Yeah. **Swyx** [27:58] And so what was the big rewrite for the 0.1 and then the 1.0? **Comfy Anonymous** [28:02] Uh, well, that's more, uh, on the front end side. That's- **Swyx** [28:05] Okay **Comfy Anonymous** [28:06] ... 'cause b-before that, it was just, uh, like the UI what-- 'Cause when I first wrote it, I just, uh, I said, "Okay, how can I make..." Like I can do web development, but I don't like doing it. **Swyx** [28:19] Mm-hmm. **Comfy Anonymous** [28:19] Like what's the easiest way I can slap a node interface on this? And then I found this library, Light Graph, like JavaScript library. **Swyx** [28:27] Life Graph? **Comfy Anonymous** [28:28] Light Graph. **Swyx** [28:29] Usually people will go for like React Flow for like a flow builder. **Comfy Anonymous** [28:31] Yeah, but that seems like too complicated **Swyx** [28:34] 'Cause of React. **Comfy Anonymous** [28:36] So I didn't really want to spend time at like developing the, the front end. So I'm like, "Well, oh, Light Graph. This has the whole node interface." So okay, let me just plug that into, to my back end then. **Swyx** [28:50] I feel like if Streamlit or Gradio offered something, you would have used Streamlit or Gradio 'cause it's Python. **Comfy Anonymous** [28:54] Yeah, Streamlit and Gradio-- Like Gradio, I don't like Gradio. It's- **Swyx** [28:58] Why? **Comfy Anonymous** [28:58] ... it's ba- Like the-- That's, that's one of the reasons why like Automatic was very bad. It's Gra- 'cause, uh, the problem with Gradio, it forces you to... Well, not forces you, but it kind of, uh, makes your, your interface logic and your back end logic and what, just sticks them together. **Swyx** [29:21] It, it is supposed to be easy for you guys for-- if you're a Python main. You know, I'm a JS main, right? **Comfy Anonymous** [29:25] Yeah. **Swyx** [29:26] If you're a Python main, it's supposed to be easy. **Comfy Anonymous** [29:27] Yeah. It's e- well, it's easy, but it makes your whole software a huge mess. **Swyx** [29:31] I see, I see. So you're mixing concerns instead of separating concerns? **Comfy Anonymous** [29:34] Well, it's 'cause- **Alessio** [29:36] Like front end and back end **Comfy Anonymous** [29:37] ... the front end and back end should be well separated with a- **Alessio** [29:40] Yeah, yeah **Comfy Anonymous** [29:40] ... defined API. Like that's, that's how you're supposed to do it. Gradio just like- **Swyx** [29:44] People, smart people disagree, but yeah **Comfy Anonymous** [29:47] ... it ju- it just stick e- sticks everything together. It makes, uh- **Alessio** [29:50] Yeah **Comfy Anonymous** [29:50] ... makes it easy to like make a huge mess. And also it's, uh... And there, there's a lot of issues with, uh, with Gradio. Like it, it's very good if all you want to do is just get, like slap a quick interface on your, uh, like to, to show off your, uh, like your ML project. Like that's what it's made for. **Swyx** [30:10] Yeah, yeah. **Comfy Anonymous** [30:11] Like, uh, like there-there's no problem using it like, oh, I have my I have my code. I just want a quick interface on it. That's perfect. **Swyx** [30:20] Mm-hmm. **Comfy Anonymous** [30:20] Like, use Gradio. But if you want to make something that's, like, a real, like, real software that will last a long time and will be easy to maintain, then I would avoid it. **Swyx** [30:33] Yeah. **Comfy Anonymous** [30:33] So. **Swyx** [30:34] So your criticism is Streamlit and Gradio are the same-- I mean, those are the same criticisms, uh- **Comfy Anonymous** [30:38] Yeah. **Swyx** [30:39] Okay. **Comfy Anonymous** [30:39] Streamlit I haven't- **Swyx** [30:41] Haven't used as much. **Comfy Anonymous** [30:41] Yeah, I've s- just, uh, looked a bit, uh- **Swyx** [30:44] Similar philosophy. **Comfy Anonymous** [30:45] Yeah, it's similar. It's just-- it, it just seems to me like, okay, for quick, like, AI demos, it's perfect. **Swyx** [30:51] Yeah. Going back to, like, the, the, the core tech, like asynchronous queues, slow re-execution, smart memory management, you know, anything that you, you, you're very proud of or was very hard to figure out? **Comfy Anonymous** [31:01] Yeah. The, the thing that's the biggest, uh, pain in the ass is probably the memory management. **Swyx** [31:06] Mm-hmm. **Alessio** [31:06] Yeah. Were you just paging models in and out, or...? **Comfy Anonymous** [31:09] Yeah. Be-before it was just, okay, load the model, completely unload it, load the new model, completely unload it. Then okay, that, that works well when your, your model are small. But if your models are big and it takes sort of like-- Let's say if someone has a, like a, A forty ninety and the model size is ten gigabytes, that can take a few seconds to, like, load, unload, load, unload. So you want to try to keep things, like, in memory, in the GPU memory as much as possible. What ComfyUI does, uh, right now is it, uh, it tries to, like, estimate, okay, like, okay, you're gonna sample this model. It's gonna take probably this amount of memory. Let's remove the models, like, this amount of memory that you load- **Swyx** [31:59] Mm-hmm **Comfy Anonymous** [31:59] ... that's been loaded on the GPU and then just execute it. But, uh, so there's a fine line between just-- 'cause try to remove the least amount of models that are already loaded 'cause as for apps like Windows driver, the-- And another problem is, uh, the NVIDIA driver on Windows by default. Because there's a way to-- There's an option to disable that feature. But by default, it, uh, like, if you start loading, you can overflow your GPU memory, and then it's-- the driver's gonna automatically start paging to RAM. But the problem with that is it's, it makes everything extremely slow. **Alessio** [32:43] Mm. **Comfy Anonymous** [32:43] So when you see people complaining, "Oh, this model, it works, but oh, shit, it starts slowing down a lot," that's probably what's happening. So y- it's basically you have to just try to get-- use as much memory as possible, but not too much, or else things start slowing down, or people get out of memory. And then just find-- try to find that line where, oh, like, the drive around window starts paging and stuff. **Swyx** [33:13] Yeah. **Comfy Anonymous** [33:13] And yeah. And the problem with PyTorch is it's, uh, it's high levels don't have that much fine grain control over, like, uh, specific, uh, memory stuff. **Swyx** [33:23] Mm-hmm. **Comfy Anonymous** [33:23] So, uh, kinda have to leave, like, the memory freeing to, to Python and PyTorch, which is-- can be annoying sometimes. **Swyx** [33:33] So, you know, I, I think one thing as a, as a maintainer of this project, like, you're designing for a very wide- **Comfy Anonymous** [33:38] Yeah **Swyx** [33:38] ... surface area of compute. Like, you even support CPUs. **Comfy Anonymous** [33:42] Yeah. Well, that's- **Swyx** [33:43] It's- **Comfy Anonymous** [33:43] ... that's just, uh- **Swyx** [33:44] For fun. **Comfy Anonymous** [33:45] For PyTorch. PyTorch CPUs. **Alessio** [33:47] Yeah. **Comfy Anonymous** [33:48] So yeah, it's just-- that's not, that's not hard to support. **Swyx** [33:50] First of all, is there a market share estimate? Like, is it, like, seventy percent NVIDIA and, like, thirty percent AMD and then, like, miscellaneous on- **Comfy Anonymous** [33:57] Uh- **Swyx** [33:57] ... Apple Silicon or whatever? **Comfy Anonymous** [33:59] Uh, for Comfy? **Swyx** [34:01] Yeah. **Comfy Anonymous** [34:01] Yeah, I'm-- Well, yeah, I don't know the market share. **Swyx** [34:04] Can you guess? **Comfy Anonymous** [34:05] Uh, I think it's mostly NVIDIA. **Swyx** [34:07] Right. **Comfy Anonymous** [34:08] 'Cause- **Alessio** [34:08] Yeah **Comfy Anonymous** [34:09] ... 'cause AM-- the problem, like, AMD works horribly on Windows. **Alessio** [34:13] Mm. **Comfy Anonymous** [34:13] Like, on, on Linux, it, it works fine. It's, it's slower than the price equivalent, uh, NVIDIA GPU. But, uh, it works. Like, you can use it, generate images, everything works on Linux. On Windows, you might have a hard time. So that's the problem. And most people-- I think most people who, uh, bought, uh, AMD probably use Windows. **Alessio** [34:38] Mm-hmm. **Swyx** [34:38] Yeah, yeah. **Comfy Anonymous** [34:38] They probably aren't, uh, gonna switch to Linux, so So u-until AMD actually, like, uh, ports their, like, RAWCM to, to Windows properly, and then there's actually PyTorch. I think they're, they're doing that. **Alessio** [34:54] Mm-hmm. **Comfy Anonymous** [34:54] Uh, they're in the process of doing that. But, uh, until they get a, they get a good, like, PyTorch RAWCM build that works on Windows, it's, uh, like, they're gonna have a hard time. **Alessio** [35:06] Yeah. **Swyx** [35:07] We gotta get George on it. ### Node Playground **Alessio** [35:08] Yeah. Well, he's trying to get Lisa Su to do one, but- Let's talk a bit about, like, the node design. So unlike all the other text to image, you have a very, like, deep-- So you have, like, a separate node for, like, clip and code. You have a separate node for, like, the K sampler. You have, like, all these nodes. Going back to, like, the making it easy versus making it hard, but, like, how much do people actually play with all the settings, you know? Kinda like how do you guide people to like, "Hey, this is actually gonna be very impactful," versus, "This is maybe, like, less impactful, but we still wanna expose it to you"? **Comfy Anonymous** [35:41] Uh, well, I try to expose, uh, like, uh, I try to expose everything or-- But, uh, yeah. At least for the-- But for things like, for example, for the samplers, like, there's, uh, like yeah, four different sampler nodes- **Alessio** [35:57] Mm **Comfy Anonymous** [35:57] ... which go in easiest to most advanced. So yeah, if you go, like, the easy node, the regular sampler node, that's-- you have just, uh, basic settings. But if you use, like, the sampler advanced-- custom advanced node, that, that one you can actually-- you'll see you have, uh- Like different nodes **Alessio** [36:20] I'm looking it up now. **Comfy Anonymous** [36:21] Yeah. **Alessio** [36:22] What are, like, the most impap- impactful parameters- **Comfy Anonymous** [36:25] Yeah **Alessio** [36:26] ... that you use? So it's like, you know, you can have more, but, like, w- which ones, like, really make a difference? **Swyx** [36:30] CFG. **Comfy Anonymous** [36:31] Yeah, they all do. They all have- ... their own, like, uh... They all, like, for example, yeah, steps. Usually you want, uh, steps, you want them to be as low as possible, but y- you want-- If you're optimizing your, your workflow, you want to... You lower the steps until, like, the images start deteriorating too much. **Alessio** [36:52] Mm. **Comfy Anonymous** [36:52] 'Cause that, um... Yeah, that's the number of steps you're, you're running the diffusion process, so if you want things to be fast, that's, uh, lower is better. But, uh, yeah, CFG, that's more... You can kind of see that as the contrast of the image. Like, if your image looks too burnt out- **Alessio** [37:12] Mm **Comfy Anonymous** [37:12] ... then you lower the CFG. So yeah, CFG, that's how... Yeah. That's how strongly the, like, the negative versus positive prompts. 'Cause when you sample a diffusion model, it's, uh, it's basically a negative prompt. It's just, yeah, positive prediction minus negative prediction. **Alessio** [37:33] Mm-hmm. **Swyx** [37:33] Contrastive loss. **Comfy Anonymous** [37:34] Yeah. So it's positive minus negative, and the CFG, that's the multiplier. **Alessio** [37:39] Yeah. **Swyx** [37:40] Yeah. **Comfy Anonymous** [37:40] That's just... Yeah, so. **Alessio** [37:41] What are, like, good resources to understand what the parameters do? I think most people start with Automatic, and then they move over, and it's like steps, CFG, sampler name, scheduler, denoise. **Swyx** [37:52] Reddit. **Comfy Anonymous** [37:54] But, uh, honestly, well, it's more-- it's something you should, like, try out yourself. **Swyx** [37:59] Mm-hmm. **Comfy Anonymous** [38:00] And I don't think you, you don't necessarily need to know how it works to- **Swyx** [38:04] Right. **Alessio** [38:04] Right. **Comfy Anonymous** [38:04] ... like what it does, 'cause even if you know, like CFG, oh, it's like positive minus negative prompt. **Alessio** [38:10] Yeah. **Comfy Anonymous** [38:11] So the only thing you know it's CFG is if it's one point zero, then that means the negative prompt is applied. It also means sampling is two times faster. But, uh- **Alessio** [38:21] Yeah. **Swyx** [38:22] Yeah **Comfy Anonymous** [38:22] ... yeah. But other than that, it's more like you should really just see what it does to the images yourself, and you'll, you'll probably get the more intuitive understanding of what these things do. **Swyx** [38:34] Mm-hmm. **Alessio** [38:34] Mm-hmm. Any other nodes or things you wanna shout out? Like, I know the AnimateDiff, IPAdapter, those are like some of the most popular ones. Um, yeah, what else comes to mind? **Comfy Anonymous** [38:44] Uh, not nodes, but there's, uh, like I... What I like is when, when some people, sometimes they make, uh, they make things that use ComfyUI as their backend. Like there's a, a plugin, uh, c- for Krita that, uh, uses ComfyUI as its backend. So you can use, uh, like all the models that work in Comfy in Krita, and, uh, I think I've tried it once, uh, but, uh, I know a lot of people use it- **Alessio** [39:14] Mm-hmm **Comfy Anonymous** [39:14] ... and find it really nice, so. **Alessio** [39:16] What's the craziest node that people have built? Like the most complicated? **Comfy Anonymous** [39:21] Craziest node, like I, I like... Yeah, I know some people have, uh, made like video games and- ... in Comfy, uh, with like stuff like that. So- ... like someone... Like I remember like, yeah, last... Think it was la- last year, someone made a, like a, like Wolfenstein 3D in Comfy. **Alessio** [39:43] Of course. **Comfy Anonymous** [39:44] And then one of the inputs was, oh, you can generate a texture, and then it- **Alessio** [39:49] Oh **Comfy Anonymous** [39:49] ... changes the texture in the game. **Alessio** [39:51] Nice. **Comfy Anonymous** [39:51] So you could plug it to like- **Alessio** [39:54] Yeah **Comfy Anonymous** [39:54] ... your workflow, and there's a lot of... If you look, there- there's a lot of crazy things people do, so. **Swyx** [39:59] Yeah. **Alessio** [40:00] And now there's like a node register that people can use to like download nodes and- **Comfy Anonymous** [40:05] Yeah, like well, there's always been the, like the ComfyUI manager. **Alessio** [40:08] Yeah. **Comfy Anonymous** [40:08] But we're trying to make this more like, I don't know, official, like, uh, with, uh, yeah, with the, the node registry. 'Cause b- before, before the node registry, the... It like, okay, how did your custom node get in the ComfyUI manager? **Alessio** [40:25] Right. **Comfy Anonymous** [40:25] That's the guy running it who like, uh, every day, he search GitHub for new custom nodes and added them manually to his, uh, to his custom node manager. So we're trying to make it, uh, a less effort for him basically. **Swyx** [40:41] Yeah. **Alessio** [40:41] Yeah, but I was looking. I mean, there's like a YouTube download node. There's like th- this is almost like, you know, a data pipeline more than like an image generation thing at this point. It's like you can get data in, you can like apply filters to it, you can generate data out. **Comfy Anonymous** [40:55] Yeah. You can do a, a lot of, uh, different things. Yeah. Something I think, uh, what I did is, uh, I made it, uh, easy to make custom nodes, so I think that, that helped a lot for- **Alessio** [41:08] Mm **Comfy Anonymous** [41:09] ... like the, the ecosystem, 'cause it is very easy. Just make a node. So- **Alessio** [41:14] Mm-hmm **Comfy Anonymous** [41:14] ... yeah, a bit too easy sometimes. Then y- then we have the issue where there's a lot of custom node packs which share similar nodes. **Alessio** [41:25] Mm. **Comfy Anonymous** [41:25] So, but, uh, well, that's, uh, yeah, something we're trying to solve by maybe bringing some of the functionality into core. **Alessio** [41:35] Yeah. ### Video Models **Swyx** [41:35] Yeah. **Comfy Anonymous** [41:35] But, uh, yeah. **Swyx** [41:37] Yeah. **Alessio** [41:38] And then there's like video. People can do video generation. **Comfy Anonymous** [41:41] Yeah. Video, that's, uh... Well, the, the first video model was, uh, like Stable Video Diffusion, which was last, yeah, exactly last year, I think. Like one year ago. But that wasn't a true video model. **Alessio** [41:54] Mm-hmm. **Comfy Anonymous** [41:54] So it was, uh- **Swyx** [41:56] As in like it was like moving images? **Comfy Anonymous** [41:58] Yeah. It generated video. What, what I mean by that is it's, uh, like, uh, it's still 2D latents. It's basically what they did is they took SD2, and then they added some temporal attention to it, and then, uh, trained it on videos. And, uh, so it's, it's kind of like animate- Diff, like- **Alessio** [42:19] Yeah **Comfy Anonymous** [42:19] ... those same, same idea basically. Why I say it's not a true video model is that you still have, like, the 2D latents. Like, a true video model like, uh, Mochi, for example, would have 3D latents. **Alessio** [42:32] Mm-hmm. **Comfy Anonymous** [42:33] So imagine- **Alessio** [42:33] Which means you can, like, move through the space basically is the, the difference. You're not just kind of like- **Comfy Anonymous** [42:38] Well- **Alessio** [42:38] ... reorienting **Comfy Anonymous** [42:39] ... so, yeah, and it's also... Well, it's also 'cause you have a temporal VAE- **Alessio** [42:42] Mm-hmm **Comfy Anonymous** [42:43] ... that also. Like, Mochi has a temporal VAE that compresses, um, like, the temporal direction also. So that's something you don't have with, like, yeah, AnimateDiff and, uh, Stable Video Diffusion. They only, like, compress it spatially, not temporally. **Alessio** [43:00] Mm-hmm. **Comfy Anonymous** [43:00] So yeah. So these models, uh, that's why I call them, like, true video models. There, there's a-- Yeah, there's actually a few of them, but, uh, the, the one I've implemented, uh, in Comfy is Mochi 'cause that, that seems to be the best one so far. **Alessio** [43:16] Yeah. We had, uh, AJ come and speak at the State of Diffusion meetup. Other open one I think I've seen is CogVideo. **Comfy Anonymous** [43:22] Yeah, CogVideo. Yeah. That one's in... Yeah, it also seems decent, but, uh, yeah. **Alessio** [43:27] Uh, Chinese, so we don't use it. **Comfy Anonymous** [43:30] No, it's fine. It's just, yeah, I could... Yeah, it's just it, uh, there's a-- It's not the only one. There's also a few others which, uh- **Alessio** [43:37] The rest are, like, cr-closed source, right? Like Kling and- **Comfy Anonymous** [43:39] Yeah. The closed source, there's a bunch of them, but I mean open. I've seen a few of them. Like, uh, yeah, I can't remember their names, but there's Cog-CogVideo is the big, uh, the big one. Then there's also a few of them that, uh, released at the same time. Uh, there's one that released at the, at the same time as SD 3.5, same day, which is why I don't remember the name. **Alessio** [44:02] Oh. We should have a release schedule, so we don't conflict on each of these things. **Comfy Anonymous** [44:06] Yeah. Yeah, I think SD 3.5 and Mochi released on the same day. So- **Alessio** [44:11] Uh-huh **Comfy Anonymous** [44:11] ... everything else was kind of drowned- **Alessio** [44:13] Mm **Comfy Anonymous** [44:13] ... completely drowned out. So, uh, for some reason, lots of people picked that day to release their stuff. Yeah. Which is, uh, well, shame for those, uh, I think, yes. I think, uh, Omnigen also released the same day- **Alessio** [44:28] Mm-hmm **Comfy Anonymous** [44:28] ... which also seems interesting, but, uh- **Alessio** [44:30] Yeah **Comfy Anonymous** [44:31] ... well. **Alessio** [44:31] Yeah. What's Comfy... So you are Comfy, and then there's, like, comfy.org. Um, I know we do a lot of things for, like, news research, and those guys also have kind of like a more open source and on, uh, thing going on. ### Comfy's Arc **Alessio** [44:42] How do you work? Like, you mentioned you mostly work on, like, the, the core piece of it, and then what- **Comfy Anonymous** [44:48] Maybe I should fill in because I, yeah, I g- Yeah, I feel like maybe, uh, yeah, I only explained part of the story. **Alessio** [44:55] Right. Yeah, yeah. **Comfy Anonymous** [44:56] Yeah. Maybe I should explain the rest. So yeah. So yeah, basically, uh, January, that's when the first-- January 2023. January 16, 2023, that's when Comfy was, uh, first, uh, released, uh, to the public. Then, yeah, did a Reddit post about the area composition thing somewhere in, uh, I don't remember exactly. Maybe end of January, beginning of February. And then someone, a YouTuber made a video about it. Uh, like, uh, Olivio, he made a video about Comfy in, uh, March 2023. I think that's when it got its real burst of attention, and by, yeah, that time I was continued to developing it, and it was getting, uh... People were starting to use it, uh, more, which unfortunately meant that I, I, my-- 'cause, uh, I had first written it to do, like, experiments, but then, well, my time to do experiments- **Alessio** [45:56] Mm-hmm **Comfy Anonymous** [45:56] ... well, started going down 'cause, yeah. 'Cause, uh, yeah, people were actually starting to use it then. **Alessio** [46:02] Right. **Comfy Anonymous** [46:03] Like, I had to-- And I say, "Well, yeah," had to add all, all these features and stuff. Uh, yeah, and then I got hired by Stability June 2023. Then I made the basically... Yeah, they hired me 'cause they wanted, uh, SDXL. So I got SDXL working very well in ComfyUI because, yeah, they were experimenting with it. Actually, the SDX-- how the SDXL release worked is they released, uh, for some reason, like, they released the code first. **Alessio** [46:34] Mm-hmm. **Comfy Anonymous** [46:35] But they didn't release the model checkpoint. Uh, yeah. **Alessio** [46:38] Oh. **Comfy Anonymous** [46:38] So they released the code, and then, well, if since the research was released the code, I, I released the code in Comfy 2. And then the checkpoints were basically early access. People had to sign up, and they only allowed, uh, a lot of people from edu emails. Like- ... if you had a edu email, like, uh, like, they gave you access basically to the zero-- SDXL 0.9, and well, that, uh, leaked. **Alessio** [47:08] Right. **Comfy Anonymous** [47:09] Of course, because of course it's gonna leak if, if you do that. Uh, well, the only way people could easily use it was with Comfy. So yeah, people started using it, and then I fixed a few of the issue that people had. So then the big 1.0 release happened, and well, ComfyUI was the only way a lot of people could actually run it on their computers. 'Cause it just, like, Automatic was so, like, inefficient and bad that, uh, most people couldn't act... Like, it just wouldn't, wouldn't work. Like, 'cause he, he did a quick implementation. So people were forced to use ComfyUI, and that's how it became popular because people had no choice. **Alessio** [47:56] The growth hack. **Comfy Anonymous** [47:57] Yeah. **Alessio** [47:57] Yeah. **Comfy Anonymous** [47:58] Yeah. Like, everywhere, like, peop-people who didn't have the 4090, they had, like, who had just regular GPUs. **Alessio** [48:04] Yeah, yeah. **Comfy Anonymous** [48:04] They ju- they didn't have a choice, so. **Alessio** [48:07] Yeah. I got a 4070, so think of me. And so today, what's-- Is there, like, a core Comfy team or...? **Comfy Anonymous** [48:14] Uh, yeah. Well, right now, um, yeah, we are hiring actually. So right now, core co- like, on the core core itself, it's, it's me Uh, but because, uh, reason we're foc- like all the focus has been mostly on the front end right now- **Swyx** [48:31] Mm-hmm **Comfy Anonymous** [48:31] ... 'cause that's the thing that's been neglected for a long time. **Swyx** [48:36] Right. **Comfy Anonymous** [48:36] So, uh, so most of the focus right now is, uh, all on the front end, but we are, uh, yeah, we will, uh, soon get, uh, more people to, like, help me with the actual backend stuff. Because that's... O- once the f- once we have our V1 release, which is gonna be the package ComfyUI with the, the nice interface and easy to install on Windows, and hopefully Mac. Yeah. **Swyx** [49:05] Yeah. **Comfy Anonymous** [49:06] Once we have that, uh, we're gonna have to l- lots of stuff to do on the backend side and also the front end side. But, uh- **Swyx** [49:14] Yeah. **Alessio** [49:15] What's the release da- I'm on the wait list. What's the timing? **Comfy Anonymous** [49:18] Uh, soo- soon. Yeah. Like, I don't want to promise a release date. **Alessio** [49:26] Yeah, yeah, yeah. **Comfy Anonymous** [49:26] Because, yeah, we, we do have a, a release date we're targeting, but yeah, I'm not sure if, uh, if it's public. **Alessio** [49:33] Yeah. **Comfy Anonymous** [49:34] Yeah. And how we're gonna... Like, we're, we're still gonna continue, like, for f- doing the open source, like making ComfyUI the best way to run, uh, like Stable Diffusion models. Like, at least the, the open source side then, like, it's gonna be best way to run, uh, models locally. **Alessio** [49:53] Mm-hmm. **Comfy Anonymous** [49:53] But we will have a few, uh, like a few things to, to make money from it, like, uh, cloud inference or, like, that type of, that type of thing. So- **Alessio** [50:04] Right **Comfy Anonymous** [50:04] ... and maybe some, like, some things for some enterprises. **Swyx** [50:09] I mean, a few questions on that. How do you feel about the other Comfy startups? **Comfy Anonymous** [50:12] I mean, I, I think it's great. **Swyx** [50:15] They're using your name, you know? **Comfy Anonymous** [50:16] Yeah. Well, it's better they use Comfy than they use something else. **Alessio** [50:19] Yeah. That's true. **Comfy Anonymous** [50:20] Yeah. Like, yeah, it's fine. I don't... Like, we're... Like, yeah, I'm-- We're gonna try not to... We, we don't want to... Like, we want them to-- people to use Comfy, 'cause, uh, like I said, it's better that people use Comfy than use something else. So as long as they use Comfy, it's, uh... I think it helps, it helps the ecosystem and stuff. 'Cause more peop- even if they don't, they, they, like... Even if they don't contribute directly, the fact that they are using Comfy means that, like, people are more likely to, like, join the ecosystem. So yeah. ### What's Next **Swyx** [50:58] And then would you ever do text? **Comfy Anonymous** [51:00] Yeah. Well, the... You can already do text with some custom nodes. So yeah, it's something we, we like. Yeah. It's something I've wanted to eventually add to core, but it's more, like, not a, not a very high priority. But, uh, 'cause a lot of people use text for, like, prompt enhancement and, like, other things like that. So it's, uh, yeah, it's just that my focus has always been, like, um, diffusion models. Yeah, unless some text diffusion model comes out. **Swyx** [51:31] Yeah. Uh, David Holtz is investing a lot in text diffusion. **Comfy Anonymous** [51:34] Yeah. Yeah. Well, if a, if a good one comes out, then well, he'll probably implant it since it fits with the whole- **Swyx** [51:39] Yeah. I mean, I, I imagine it's gonna be closed source to Midjourney, so. **Comfy Anonymous** [51:43] Yeah. Well, if a, yeah, if an open one comes out. **Swyx** [51:48] Yeah. **Comfy Anonymous** [51:49] Yeah. Then, uh, yeah, I'll probably, yeah, yeah, I'll probably implement it. It's, uh- **Alessio** [51:55] Cool, Comfy. Thanks so much for, for coming on. This was fun. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.