An Operational Scorecard for Evaluating Enterprise AI Agents on Data Platforms: A Reproducible Test Protocol for Reliability, Governance, Security, Recovery, and Cost

Enterprise AI agents increasingly combine language models, retrieval systems, tools, and data-platform permissions to complete multi-step work. Evaluating these systems using task success alone can conceal failures involving unauthorized access, missing evidence, unsafe retries, and excessive cost. This paper proposes an operational scorecard and reproducible test protocol for evaluating enterprise AI agents across six dimensions: task correctness, authorization and security, evidence and lineage, operational reliability, recovery behavior, and efficiency. It defines a weighted scoring formula, non-compensable safety gates, a reusable test-case schema, and twelve proposed data-platform evaluation scenarios. The framework complements the author's related work, Agentic AI: A Reference Architecture for Governed Enterprise AI Agents on the Lakehouse (https://doi.org/10.5281/zenodo.22807070), by focusing on evaluating agent behavior rather than architectural design alone. This publication presents a proposed framework, not a completed empirical benchmark. Numerical examples are hypothetical, and implementation and independent validation remain future work.