Accelerating Deep-Learning Applications with Efficient Tensor Layout Management

Deep learning applications use tensors, the fundamental data structures, for storing input/output data and managing intermediate weights and parameters during training/inference. The memory layout of tensors can be changed from contiguous to non-contiguous by tensor operators at runtime. In this paper, we comprehensively study the impact of tensor memory layout on the performance of deep-learning applications. The study shows that performing conversions at runtime yields speed improvements of 8.17% − 249.6%. Based on this observation, we propose a new approach, SmartConvert, which is designed to dynamically convert the memory layout of intermediate tensors from non-contiguous to contiguous and hide the conversion time by overlapping its cost with I/O or communication operations during training and inference. We conduct a preliminary performance study using GAT models. The results show that SmartConvert can reduce the computation time by 23% on average.