An Overview of Multilevel Co-Design for AI Accelerator: From Fundamentals to the Frontier

The breakdown of Moore’s Law has shifted deep learning acceleration from compute-bound to memory-bound optimization, necessitating comprehensive co-design across multiple abstraction layers. This analysis examines key co-design methodologies spanning the algorithm-architecture, architecture-physics, and integration-system domains for modern AI hardware, addressing the limitations of conventional silicon approaches. CNNs match spatial hardware arrays with tailored dataflow mapping, while GNNs manage irregular access patterns. Similarly, LLM accelerators embed key-value cache optimizations and hierarchical compression directly into the silicon, and edge inference systems adapt to physical constraints by jointly tailoring quantization at the hardware-software boundary. Furthermore, architecture and physics co-design methods, including processing-in-memory architectures, non-volatile memory interfaces, and neuromorphic computing, directly challenge von Neumann bottlenecks by integrating computation with storage at the circuit level. Chiplet ecosystems, 3D stacking, and advanced interconnect architectures enable system scaling beyond single-die limitations, with thermal management and power delivery emerging as critical factors in post-Moore hardware deployment. This survey presents a multi-level co-design perspective for AI accelerators, unifying algorithmic, architectural, and system layers through a cross-layer taxonomy and a general analytical model for computation and data movement. By synthesizing quantitative evidence across state-of-the-art designs and outlining key open challenges, it provides a structured overview of modern accelerator trends and the remaining technical gaps in the field.