H3D-LLM: Heterogeneous 3D Chiplet Design for LLM Inference with Dynamic Task Scheduling and Memory-Aware Orchestration

The exponential growth of Large Language Model (LLM) intensifies hardware demands for energy-efficient, low-latency architectures with scalable memory bandwidth. While 3D chiplet integration addresses conventional systems’ memory wall limitations, three critical challenges persist: asymmetric compression constraints from divergent sparsity-precision requirements across attention/projection layers, tier-level load imbalance from static resource allocation in dynamic computation patterns, and coupling-induced signal degradation in high-density TSV networks, especially under LLM-phase-specific traffic with spatiotemporal burstiness. To address these, we present H3D-LLM, a vertically heterogeneous architecture combining analog/digital Computing-in-Memory (CIM) and Neural Processing Unit (NPU) chiplets through three innovations. First, a Sparse-Aware Dynamic Execution Framework (SADEF) with Precision-Adaptive Quantization Mechanism (PAQM) enables hardware-aware compression via layer-wise unstructured sparsity detection and INT4/8-FP/BF16 mixed precision. Second, a 3D Spatio-Temporal Interleaved Parallelism (3D-STIP) with Semantic-Aware Tiered Storage (SATS) eliminates resource imbalance and improves memory efficiency through dynamic sub-batch partitioning and Key-Value (KV) cache aware management. Third, a Phase-Adaptive TSV Management (PATM) scheme with dynamic encoding and cluster-based allocation enhances inter-connect efficiency and signal integrity through runtime-aware partitioning and phase-specific dataflow scheduling. Evaluations on Llama-7B demonstrate that H3D-LLM achieves 12.3× higher energy efficiency and 8.4× faster inference than the A800 GPU, while its TSV strategy increases eye height by 12% and reduces bit error rate by up to 60× compared to naïve 3D accelerators.