Focus Session: Hardware/Software Co-Design to Accelerate Generative AI Workloads on Heterogeneous Architectures

Effective acceleration of Generative AI workloads is constrained by heterogeneous computational and memory characteristics. Large Language Models (LLMs), for example, are composed of diverse kernels, ranging from memory-bound attention mechanisms to compute-intensive feed-forward networks. Such variability leads to poor resource utilization and renders homogeneous architectures inefficient. This paper outlines a methodology to address this challenge using two synergistic approaches. First, we discuss the design of a three-dimensional (3D) manycore architecture enabled by heterogeneous integration. Specifically, the architecture leverages emerging non-volatile memory alongside CMOS-based computing cores to create an optimized kernel-to-compute mapping. Second, we explore a hardware and software co-design framework to enable fine-tuning of unimodal and multimodal LLMs. This framework ensures thermally robust inference when leveraging emerging non-volatile memories within the heterogeneous architecture. Experimental results on multiple LLMs demonstrate the efficacy of the 3D heterogeneous architecture and the co-design framework.