DynaSpa: Exploiting Spatial Sparsity for Efficient Dynamic DNN Inference on Devices

Recent advancements in exploring machine learning models' dynamic spatial sparsity have demonstrated great potential for superior efficiency and adaptability without compromising accuracy when compared to conventional static-and-dense DNNs. However, realizing theoretical inference acceleration under practical deployment environments is still faced with significant system challenges. Current vendor libraries and tensor compilers fall short due to their extra data copy operations or insufficient computation schemes, especially for DNN operators with dynamic spatial sparsity. To bridge this gap, we propose DynaSpa, an automated kernel generation framework that enables efficient on-device inference for DNNs with dynamic spatial sparsity across diverse computing platforms. DynaSpa jointly optimizes computation and sparse patterns, while also leveraging the underlying hardware characteristics. DynaSpa consistently outperforms state-of-the-art vendor libraries and tensor compilers on embedded and mobile GPUs. For DNN operators with spatial sparsity ratio between 50% ~ 90%, DynaSpa achieves a speedup of X1.3 ~ X4.4 for Jetson AGX Orin GPU, X1.6 ~ X7.7 for Jetson AGX Xavier GPU, and X1.5 ~ X7.8 for Adreno mobile GPU, when compared to their respective dense counterparts.