RoFormer: Enhanced Transformer with Rotary Position Embedding

A novel method named Rotary Position Embedding(RoPE) is proposed to effectively leverage the positional information in transformer-based language models and enables valuable properties, including the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping the linear self-attention with relative position encoding.