November 2024 Summaries

ComfyUI Desktop, previously known as V1, has been open-sourced and is now available for beta testing on Windows (NVIDIA) and macOS (M series), though it is not yet stable enough to fully replace existing setups. The application features an onboarding experience for importing existing settings, integrated logs and terminal for easier debugging, and a Help Menu with useful actions like accessing directories and logs. Users can opt-in to send crash reports for debugging, which exclude personal information or workflows. The server configuration is integrated into the app, allowing easy customization, and ComfyUI-Manager is now part of the core, with plans to launch a Registry to establish long-term standards for the ecosystem. The team is committed to ongoing improvements in stability and user experience while encouraging feedback and participation from the community.

ComfyUI has introduced support for new Stable Diffusion 3.5 Large ControlNet models by Stability AI, which includes Blur, Canny, and Depth models, each boasting 8 billion parameters and available for both commercial and non-commercial use under the Stability AI Community License. These models enhance image generation by offering detailed and customizable outputs: the Blur model excels in high-fidelity upscaling for resolutions up to 16K, the Canny model uses edge maps for structured illustrations, and the Depth model employs depth maps for precise compositions in applications like architectural renderings. Users are encouraged to explore these models using the all-in-one SD3.5 large checkpoint, with workflows available for each model to facilitate experimentation and creative exploration. Additional ControlNet models and new control types are being developed, providing further opportunities for innovation and customization in image generation.

LTX Video (LTXV), a cutting-edge video generation model developed by Lightricks, is now natively supported in ComfyUI, allowing users to generate high-quality videos in real-time. This model, based on a 2-billion-parameter DiT architecture, can produce 24 FPS videos at a 768x512 resolution faster than they can be watched, specifically in just four seconds for a five-second video. LTXV is optimized for accessible GPUs, such as the RTX 4090, and uses bfloat16 precision to ensure efficient memory usage without compromising quality. The model's Diffusion Transformer architecture guarantees smooth motion and avoids common problems like object morphing, making it ideal for creators aiming to produce consistent long-form videos. LTXV's native integration into ComfyUI includes custom nodes branded as "LTXVideo," which are easily accessible through the ComfyUI Manager, offering additional functionalities such as image-to-video conversion. The introduction of LTXV in ComfyUI opens new avenues for creators to explore real-time video generation and storytelling possibilities.

ComfyUI has integrated support for a new series of models from Black Forest Labs designed for Flux.1, including the Redux Adapter, Fill Model, ControlNet Models, and LoRAs (Depth and Canny). These models enhance the precision and control over details and styles in image generation. The Fill Model facilitates inpainting and outpainting by using input images, masks, and prompts to fill or extend images seamlessly. The Redux model generates image variations without prompts, maintaining the style, color palette, and composition of the input image. ControlNet models provide specific conditioning via depth and canny edge maps, offering enhanced control over image structure and design. Users are encouraged to update ComfyUI, download the necessary models, and utilize example workflows to explore these new capabilities.

ComfyUI v0.3.0 introduces significant enhancements to its interface, focusing on efficiency and user experience, with a newly redesigned native reroute system that improves performance and reduces rendering overhead while making workflow files smaller. The update also enhances group management by allowing smart selection and hierarchical nesting, facilitating cleaner and more manageable complex workflows. Users can now monitor backend server logs without leaving their workflow, thanks to an integrated log viewer. A new intelligent view management system provides automatic adjustments for smoother transitions and a full canvas view. Additionally, the refreshed UI has become the default, although users can revert to the classic UI if preferred. These updates aim to streamline workflow creation, making it faster and more intuitive, and are showcased in demo videos to highlight their practical application.

ComfyUI has introduced optimized support for Genmo's latest video generation model, Mochi, allowing users to achieve high-quality video outputs on consumer-grade GPUs like the 4090. The Mochi 1 model, available in 480P with an HD version expected later, is released under the Apache 2.0 license, offering flexibility for developers to use and modify it without restrictive licensing. The integration features multiple attention backends to efficiently utilize under 24GB of VRAM. Users can easily run Mochi on ComfyUI by updating to the latest version, downloading the necessary weights, and following the provided workflow steps. For those with limited RAM, alternatives such as the fp8_scaled models are suggested. Additionally, a packaged checkpoint simplifies the setup by including necessary components, facilitating a streamlined video generation process.