ALN: Approximate Layer Normalization for Transformer Training on Edge Device

Transformers have achieved significant success across various fields, and training Transformers efficiently on resource-constrained platforms with private user data has been attracting attention for better domain adaptation while enhancing data security. However, among the components of Transformers, Layer Normalization (LN) and its hardware implementation have received limited attention, constraining the overall efficiency of on-device Transformer training. To address both the algorithmic and hardware challenges associated with deploying LN on Transformer training accelerators, this work proposes an Approximate Layer Normalization (ALN) algorithm and its hardware design. At the algorithmic level, the mean and variance are approximated using a small number of input features, significantly reducing computational and memory access overhead, while decoupling data dependencies during both forward propagation (FP) and backward propagation (BP) phases. At the hardware design level, a resource-sharing strategy is employed to reduce hardware overhead, and an efficient scheduling method is proposed to improve throughput. By adopting the algorithm-hardware co-design approach, ALN achieves an improved trade-off between hardware efficiency and accuracy. Experimental evaluations on the ViT-B16, Swin-T, and ALBERT models demonstrate that ALN maintains accuracy while significantly improving computational efficiency. Compared with state-of-the-art designs, ALN reduces operations by 38.3% and memory accesses by 29%. The ALN hardware is implemented on the Xilinx ZCU102 field-programmable gate array (FPGA). Hardware implementation results show that the proposed ALN module reduces digital signal processor (DSP) usage by 46.7% and power consumption by 27% compared with the conventional LN design, while improving logic usage efficiency, DSP usage efficiency, and energy efficiency by 1.93$\times$mml:mathmml:mo×, 4.41$\times$mml:mathmml:mo×, and 3.22$\times$mml:mathmml:mo×, respectively.