Spine-Free Networks for Large Language Model Training

We study the optimal parallelization strategy of large language models (LLMs) and demonstrate that LLM training workloads generate sparse communication patterns in the network. Consequently, we argue that LLM training clusters do not require any-to-any full-bisection networks. We then propose Rail-only, a novel datacenter network architecture tailored to LLMs’ unique communication patterns. Rail-only networks eliminate the spine layer of conventional fabrics, resulting in lower cost and energy consumption. We demonstrate that Rail-only networks achieve the same training performance while reducing network cost by 38% to 77% and network power consumption by 37% to 75%, compared to traditional GPU clusters with full-bisection bandwidth.