Too Good to Be True? Generalization Challenges in LLM-Based Fake News Detection

Although recent studies report high accuracy in fake news classification using supervised models, such models often rely heavily on spurious correlations in training datasets and fail to generalize to unseen contexts. A smaller body of work has explored more realistic approaches based on linguistic cues, such as subjectivity, aiming to reduce the reliance on dataset-specific content learned during training. In this study, we assess the capabilities of four state-of-the-art large language models (LLMs) — Llama3.1-70B, Claude 3.5 Sonnet, GPT-4o, and Mistral Large 2 — in identifying fake news based solely on linguistic features, without access to external verification. Two experimentswere conducted: one with generic prompts and another with fine-grained instructions and examples, including classification of false excerpts into categories like untrue facts, exaggerations, and incorrect named entities. Despite the well-known linguistic capabilities of modern LLMs, our results show consistently low performance across both experiments. This outcome supports the hypothesis that detecting disinformation based solely on content or linguistic cues, without external factual grounding, is far more challenging than commonly reported, and that current evaluation protocols may overestimate the generalization capabilities of fake news classifiers.