Performance Comparison of the LLM Models on LLM-Based Seed Generation Method for IoT Device Fuzzing

The proliferation of IoT devices has increased the need for device security measures. Recently, an method of initial seeds generation utilizing an LLM (Large Language Model) for IoT device fuzzing (LLM-based seed generation method, in short) has been proposed to reduce the cost and improve the efficiency of vulnerability detection. However, various LLMs have been released in recent years, and existing studies have not adequately evaluated their adaptability to various LLM models. In this paper, we compare the seeds generation and the associated fuzzing results by the LLM-based seed generation method among the various LLM models of Vicuna v1.5, Mistral, Phi-3, and Llama 3. Comparison experiment results show that Mistral, Phi-3, and Llama 3 detected many vulnerabilities with high conformance rates of the seed format.