Tensor Parallelism (TP) in Transformers: 5 Minutes to Understand

Tensor Parallelism (TP) is a technique used to efficiently distribute the computational workload of transformer models across multiple GPUs, addressing the challenges posed by the increasing size of these models. It involves splitting matrix multiplications into parallel tasks, either through column-parallel or row-parallel approaches, to allow each GPU to compute a portion of the workload independently. In transformers, TP is applied to the Multi-Head Attention (MHA) and Feed-Forward Network (FFN) components, with specific strategies for dividing the projection matrices and attention heads among the GPUs to minimize the need for inter-GPU communication. Despite its advantages in reducing memory usage per GPU, TP has constraints, such as the requirement that the number of attention heads and the feed-forward hidden dimension be divisible by the number of GPUs. It also does not address all scalability challenges, as it is limited by the number of attention heads and can suffer from degraded performance due to the communication overhead, particularly across multiple nodes. TP can be implemented in practice using the Hugging Face Transformers library, although additional parallelism techniques like Pipeline Parallelism may be necessary for further optimization.