Functional Interpolation for Relative Positions Improves Long Context Transformers

It is theoretically prove that this can represent some of the popular relative position encodings, such as T5’s RPE, Alibi, and Kerple, and empirically show that FIRE models have better generalization to longer contexts on both zero-shot language modeling and long text benchmarks.