Agent-Judge-Framework: A Reliability-Oriented Framework for Automated Evaluation of Tool-Using Autonomous Agents

AJF (Agent-Judge-Framework) is a reliability-oriented framework designed to evaluate tool-using autonomous agents while simultaneously assessing the reliability of the evaluation process itself. The architecture consists of four interconnected layers: - Layer 1: Knowledge & Evidence Base - Layer 2: Agent Execution (Direct, RAG, Tool-Using) - Layer 3: Independent Judge (Zero-Shot, Structured Evidence) - Layer 4: Reliability Assessment (Bias, Calibration, Agreement) The experimental design compares three agent configurations, two independent LLM judges (via Groq API), and two evaluation strategies. Five hypotheses are tested: - H1 (Evidence Grounding): Supported - H2 (Position Bias): Bias = 93 points - H3 (Judge Reliability): Partially Supported - H4 (Calibration): ECE = 0.0528 (after Temperature Scaling) - H5 (Robustness): Partially Supported Resources: - Zenodo DOI: 10.5281/zenodo.23907823 - Figshare DOI: 10.6084/m9.figshare.34884414 - PyPI: https://pypi.org/project/agent-judge-framework/ - Website: https://ajf-core.netlify.app - GitHub: https://github.com/gitdeeper15/ajf License: MIT (code) + CC-BY-4.0 (manuscript, figures, documentation). Author: Samir Baladi (ORCID: 0009-0003-8903-0029)