EdgeQAT: Entropy and Distribution Guided Quantization-Aware Training for the Acceleration of Lightweight LLMs on the Edge

The entropy and distribution guided QAT is proposed for the optimization of lightweight LLMs to achieve inference acceleration on Edge devices and a token importance-aware adaptive method to dynamically quan-tize the tokens with different bit widths for further optimization and acceleration is designed.