Comparisons Between Open-LLMs For Fake News Detection

The spread of fake news on social media is a growing concern with significant negative societal impacts. To address this challenge, researchers have proposed various methods and algorithms for fake news detection. Recently, large language models (LLMs) have emerged as promising tools for both detecting and explaining fake news. This paper presents results from classification and explanation of true and fake news using three LLMs—Mistral, LLaMA3, and LLaMA4—applied through zero-shot prompting via Together.ai. The models analyzed 1,228 claims from the Fin-Fact dataset, evenly divided between real and fake news. Our findings are threefold. First, none of the three LLMs achieved strong classification performance, with LLaMA4 recording F1-score of 0.59 for fake news and 0.51 for true news. Second, in terms of explanation quality, LLaMA4 outperformed the other models, achieving the highest METEOR score (0.072), also obtained the lowest negative COMET score (–0.260) demonstrating improvement over its predecessor, LLaMA3. Third, all models consistently performed better on fake news than on true news, a counterintuitive outcome, since fake news is generally considered more challenging to interpret. This consistency across metrics, METEOR, BERTScore F1 and COMET, reinforces the validity of this observation. The relatively high BERTScore F1 (0.6+) indicated that all models produced semantically consistent explanations. However, both METEOR and COMET scores were very low relative to their respective scales, suggesting that the LLMs’ justifications differed in wording and judgment from those in Fin-Fact. Those metrics are designed for short and precise texts and are unsuitable for evaluating the long narrative justifications present in the Fin-Fact dataset.