Ulysses Unbound: Experiments in Communication–Computation Overlap

Ulysses Unbound explores innovative approaches to optimize video diffusion models' performance as sequence lengths increase, focusing on context parallelism and the Ulysses method, which efficiently maps to modern GPU clusters for high-throughput communication. The core idea involves sharding full sequences and leveraging dense attention at large context lengths, with a straightforward execution flow encompassing QKV projection, pre-attention communication, and attention computation. Async Ulysses improves upon baseline performance by overlapping communication and computation, yielding significant reductions in latency. Further optimization is achieved with Symmetric Memory and Fused QKV Projections, which reduce kernel launches and communication exchanges, providing substantial performance gains, particularly at lower GPU scales. The study concludes by suggesting that while no single strategy is universally superior, a dynamic runtime policy could optimize between packing, overlap, and fusion based on specific parameters, with communication-heavy workloads potentially benefiting the most.