Deciphering Digital Detectives: Understanding LLM Behaviors and Capabilities in Multi-Agent Mystery Games.

In this study, we explore the application of Large Language Models (LLMs) inJubensha, a Chinese detective role-playing game and a novel area inArtificial Intelligence (AI) driven gaming. We introduce the first datasetspecifically for Jubensha, including character scripts and game rules, tofoster AI agent development in this complex narrative environment. Our workalso presents a unique multi-agent interaction framework using LLMs, allowingAI agents to autonomously engage in this game. To evaluate the gamingperformance of these AI agents, we developed novel methods measuring theirmastery of case information and reasoning skills. Furthermore, we incorporatedthe latest advancements in in-context learning to improve the agents'performance in information gathering, murderer identification, and logicalreasoning. The experimental results validate the effectiveness of our proposedmethods. This work aims to offer a novel perspective on understanding LLMcapabilities and establish a new benchmark for evaluating large languagemodel-based agents.