Timestep distillation: 2.5x faster FLUX.2 image generation

Diffusion models, while revolutionary in image generation, face challenges with computationally expensive iterative sampling processes, making real-time generation difficult. Timestep distillation emerges as a solution by compressing the sampling process from typically 20 steps to 4-8 steps, offering 2-3x speedups without compromising image quality. This approach cannot be combined with DiT cache due to differing assumptions about timestep outputs in distilled models. The study explores applying timestep distillation techniques to the FLUX.2 model using Distribution Matching Distillation (DMD), which trains a student model to match the teacher model's sample distribution rather than individual denoising steps. Key to this process is the two-timescale update rule (TTUR) and a GAN discriminator, which provide stability and sharpened image quality through adversarial supervision. Engineering optimizations utilizing NVIDIA's FastGen framework and techniques like mixed precision training, FSDP, and activation checkpointing allow the distillation process to handle the large memory demands of models like FLUX.2. As a result, FLUX.2 achieves a 2.5x speedup with minimal quality degradation, illustrating the potential of these techniques for efficient image generation.