Behavioural Analysis of AI-Generated Phishing Emails Using Generative Models

Generative artificial intelligence (AI) systems are empowering a new generation of increasingly sophisticated phishing attacks, as cybercriminals use this technology to generate phishing emails that are more difficult for both people and automated systems to distinguish from legitimate communication. In this paper, we provide a behavioral analysis of a collection of phishing emails generated by a fine-tuned GPT-2 language model. We investigated several salient linguistic and structural properties, such as email length, perplexity, similarity to known phishing content, and grammatical accuracy, to identify distinctive patterns specific to AI-generated phishing emails. Leveraging datasets of non-AI-generated phishing emails, we find that differences in these behavioral properties are consistent and can thus be used to discriminate against AI-generated samples. In particular, we show that AI-generated emails are shorter, more grammatically accurate, moderately like real phishing attempts, and less linguistically complex. These behavioral signatures form the basis for a detection system that can be used to adaptively counteract AI-driven phishing attacks in practice.