Transformer-XL: Language Modeling with Longer-Term Dependency

A novel neural architecture, Transformer-XL, is proposed for modeling longerterm dependency and introduces a notion of recurrence by reusing the representations from the history to address the limitation of fixed-length contexts.