M3DKV: Monolithic 3D Gain Cell Memory Enabled Efficient KV Cache & Processing

Transformer-based generative large language models (LLMs) have revolutionized natural language processing, yet their quadratic growth in computational complexity in context length creates severe inference bottlenecks. While LLM keyvalue cache (KV cache) enhances decoding efficiency, prolonged contexts infer frequent KV cache reloads that exacerbate memory bandwidth constraints. To address this hardware challenge, we propose M3DKV-a monolithic three-dimensional (3D) gain cell near-memory computing accelerator featuring back-end-of-line (BEOL) cache layers for in-situ KV matrix buffering and computation and a front-end-of-line (FEOL) base layer for full selfattention operations. Through optimized 3D data organization, inter-layer dataflow management, and intelligent computation scheduling, our design achieves $0.29 \mathrm{~TB} / \mathrm{s} /$ core on-die bandwidth while demonstrating $97.03 \times / 268.01 \times$ speedup over GPU/CPU in the decoding stage and $1.72 \times-262.16 \times$ better area efficiency per parameter versus state-of-the-art accelerators.