Human‐Agent Teaming on Intelligence Tasks ( <scp>HATIT</scp> ): A Testbed for Evaluating <scp>AI</scp> in Intelligence Analysis

ABSTRACT Artificial intelligence (AI) has been proposed to overcome distributed team cognition and information challenges (e.g., volume, velocity) in intelligence analysis. However, before deploying AI in the workplace, designers should evaluate the effects of proposed AI in realistic simulated environments, also known as testbeds. We conducted interviews with intelligence professionals which, combined with our research needs, resulted in requirements, designs, and the creation of the Human‐Agent Teaming on Intelligence Tasks (HATIT) testbed. HATIT includes a web‐based software platform, a shift handover intelligence task in a fictional world with 427 pages and 60 documents, and an initial, static AI agent called “Illuminate” that summarizes documents and provides social media topic models. HATIT enables controlled experiments in a realistic simulation. This testbed lets us evaluate the effects of different AI on perceptions (e.g., trust, workload), problem solving and team cognition, and information search before deploying in actual work settings with real consequences.