December 2022 Summaries

Hungry Hungry Hippos: Towards language modeling with state space models``` State space models have demonstrated strong sequence modeling performance, but underperform compared to attention-based models like Transformers in language modeling. This is due to poor hardware utilization and inefficiencies in training SSMs. To address this, researchers propose a new state space model layer called H3 that improves recall of earlier tokens and comparison across the sequence, achieving comparable performance to Transformers on certain tasks. Additionally, they introduce FlashConv, a novel algorithm that improves efficiency by up to 2x speedup on long-range tasks, allowing for larger models to be trained at lower costs. The use of these advancements enables the creation of hybrid language models that outperform Transformers in zero- and few-shot learning scenarios and achieve better perplexity on certain benchmarks.

The AQ-SGD algorithm provides a method for reducing communication bottlenecks in decentralized training by compressing activations, which can lead to significant speedups without compromising model quality. The algorithm is designed to work with pipeline parallelism and has been shown to achieve competitive convergence rates with vanilla SGD. Theoretical analysis shows that the convergence rate of AQ-SGD is O(1/√T) for non-convex objectives under standard assumptions. Empirical studies demonstrate that AQ-SGD can tolerate aggressive quantization without compromising model quality, achieving up to 4.3x speedup over no-compression baseline and 8.5x when combined with QuantizedAdam. Future directions include exploring the effectiveness of AQ-SGD in pre-train workflows and optimizing precision and scheduling jointly.