QLlama: An FPGA-Based Microscaling Quantization Accelerator for Energy-Efficient Llama2 Inference

To address the computational power and energy efficiency challenges in Llama2 large-model inference, this letter proposes a hardware-software co-design method and finally implements a high energy efficiency accelerator named QLlama based on FPGA. This work first employs a novel quantization method based on a microscaling data format, which allows sharing a scaling factor with E8M0 format for each subtensor block, thus enabling quantization and dequantization operations to be completed using only shift operations. Second, on this basis, a mixed precision configuration is implemented for different layers of Llama2 to balance accuracy loss and computational efficiency. Finally, a dedicated accelerator QLlama is designed, whose core units include a quantization unit for dynamic quantization, a vector-matrix multiplication unit for high density computation of quantized weights, a scaled dot product unit, and a basic operator unit. Experimental results show that this scheme achieves energy efficiency improvements of $2.13\sim 10.66\times$ with negligible accuracy loss, i.e., <0.2 perplexity increase. The code is available at https://github.com/wendadawen/QLlama.