A-IO: Adaptive Inference Orchestration for Memory-Bound NPUs
A-IO: Adaptive Inference Orchestration for Memory-Bound NPUs
Chen Zhang,Yan Ding,3 Authors,Kenli Li
TLDR
The ``Model Scaling Paradox'' is revealed, and the kernel synchronization overhead of fine-grained speculative decoding under NPU computational graph compilation, and the severe limitations of purely relying on micro-level acceleration algorithms like Prompt LookUp Decoding (PLD) are pointed out.
Abstract
During the deployment of Large Language Models (LLMs), the autoregressive decoding phase on heterogeneous NPU platforms (e.g., Ascend 910B) faces severe memory-bound challenges. This study reveals the ``Model Scaling Paradox''caused by the static deployment of single-sized models. It also points out the kernel synchronization overhead of fine-grained speculative decoding \cite{leviathan2023fast, chen2023speculative} under NPU computational graph compilation, and the severe limitations of purely relying on micro-level acceleration algorithms like Prompt LookUp Decoding (PLD)