A Comparative Analysis of BERT and LLM Models for Advanced Phishing Detection

Phishing attacks remain one of the most persistent and rapidly evolving threats in cybersecurity, increasingly fueled by AI-generated malicious content. This paper presents a focused comparative analysis of two advanced language model-based techniques for phishing email detection: a fine-tuned transformer model—RoBERTa, a variant of BERT— and a locally deployed Large Language Model (Llama 3.2 1B) utilizing prompt engineering. To ensure fairness and consistency, the study employs a curated, balanced subset of a comprehensive phishing email dataset comprising 1,000 emails (624 legitimate, 376 phishing). The transformer based approach fine-tunes a pre-trained RoBERTa model for binary classification, while the LLM operates in a zero-shot setting using structured prompts, requiring no additional training. Performance evaluation using standard classification metrics reveals that the RoBERTa-based model significantly outperforms the LLM, achieving 95.0% accuracy and a phishing F1-score of 92.9%. In contrast, the LLM attains only 67.3% accuracy and a 57.4% F1-score. RoBERTa also demonstrates stronger discrimination (ROC-AUC 0.9380 vs. 0.6594) and near-zero false positives, reinforcing its reliability for practical deployment.