Building LLM-based AI Agents in Social Virtual Reality

In this paper, we introduce the design and evaluation of an LLM-based AI agent for human-agent interaction in Virtual Reality (VR). Our AI agent system leverages GPT-4, a Large Language Model (LLM) to simulate human behavior. Our LLM-based agent, deployed in VRChat as a Non-playable Character (NPC), exhibits the ability to respond to a player by providing context-relevant responses followed by appropriate facial expressions and body gestures. Our preliminary evaluation yielded the most optimal parameters for generating the most plausible responses. With our system, we lay the groundwork for future development of LLM-based NPCs in VR.