Improve the Utility of Tensor Cores by Compacting Sparse Matrix Technique

Neural networks have demanding computational requirements, particularly in matrix multiplication operations. To address this challenge, we propose a model that combines network pruning and matrix compression techniques. Our approach leverages NVIDIA's tensor cores, which excel at efficient matrix operations. We compress the network weights based on the tensor core structure and perform convolutions using the compressed weight matrix on the tensor cores. Our model incorporates neural network pruning, mixed-precision training, and compression of network weight tensors using the im2col algorithm and CSR format. We also utilize tensor kernels with a block size of 16×16 for multiplication. We evaluate the performance of various models, including pruned, AMP-optimized, combined pruning, and AMP techniques, as well as our proposed model. Our evaluation reveals a significant performance improvement compared to a simple baseline model. Through an extensive analysis of related works, we establish foundational concepts, present our proposed model, and share the obtained results.