A Novel Black-Box Evaluation Method for Large Language Models Based on Generative Adversarial Networks

Abstract: Large Language Models (LLMs) employ advanced safety alignment mechanisms designed to prevent the generation of harmful or malicious content. Despite these protections, LLMsremain susceptible to a range of security vulnerabilities that can be exploited by adversaries usingvarious attack techniques. In this work, we introduce a novel evaluation method based on automatedblack-box adversarial attack approach, designed to assess and probe LLMs without requiring accessto their internal architecture. Our method leverages a characteristic failure mode commonly observed during GAN training, known as mode collapse. Specifically, we train an enhanced SequenceGenerative Adversarial Network (SeqGAN) on the proposed data set of 808,700 questions until thegenerator enters a collapsed state, producing highly junk sequences. We refer to this resulting modelas J-SeqGAN (Junk generated by Sequences-GAN), highlighting its generation of ”junk” sequencesthat form the basis for subsequent adversarial attacks. We show that these sequences function asadversarial noise, implicitly optimized through the adversarial dynamics of GAN training. They canbe employed to evaluate the robustness of Large Language Models, specifically their ability to resistexploitation and prevent the generation of harmful or malicious content.Keywords: Large Language Models, Generative Adversarial Networks, Adversarial noise,Black-box testing, Jailbreak.