When Not to Trust Language Models: Investigating Effectiveness and Limitations of Parametric and Non-Parametric Memories
When Not to Trust Language Models: Investigating Effectiveness and Limitations of Parametric and Non-Parametric Memories
Alex Troy Mallen,Akari Asai,3 Authors,Daniel Khashabi
2022 · DOI: 10.48550/arXiv.2212.10511
arXiv.org · 152 Citations
TLDR
It is shown that LMs struggle with less popular factual knowledge, and that scaling fails to appreciably improve memorization of factual knowledge in the tail, and that retrieval-augmented LMs largely outperform orders of magnitude larger LMs, while unassisted LMs remain competitive in questions about high-popularity entities.