This paper is included in the Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation.

Alpa automates model-parallel training of large deep learning models by generating execution plans that unify data, operator, and pipeline parallelism and generalizes to models with heterogeneous architectures and models without manually-designed plans.