Drift Testing for Conversational AI: What Changed for the Customer, Build After Build
A conversational AI product changes on every build: a new model, a new prompt, new documents, a new retrieval setting. The same question gets a differently worded answer, and a tester signing off a release needs to know one thing: did the answer change in a way that matters to the customer? This paper describes Behavior Drift Analysis, a method for drift testing, which is regression testing for AI applications: it expects answers to change between builds and looks for the changes that matter. A New Answer is checked against three references: the Baseline a tester has already accepted, the Project Knowledge documents the application is meant to draw on, and the Expectations the tester has written in plain language. Every change is read three ways: facts (numbers, conditions, negation, obligations), care (what the answer does for the customer: apologizes, reassures, gives a next step, or presses and sells) and certainty (how sure the answer sounds), and each finding that matters carries a Customer Impact line in the customer's terms. Rules decide first; a small natural language inference model, run in both directions on the pairs the rules form, decides what rules cannot, and its reading stays on every finding. No large language model runs at test time; the analysis runs on the tester's own computer and is reproducible on a recorded pipeline signature. The tester's judgment is a record of its own: accepted findings with a reason and an expiry, and a release gate that respects them. The paper states the method and its formal core, how the method is itself tested, and its limits, and it introduces Build Drift Bench, a public benchmark of 260 answer pairs in four languages, with a protocol for measuring any implementation on it. It reports no results of the method. This document describes the method. Parts of it are implemented in the preview release of the Ai4Testers software; the software's own documentation states which. AI4TESTERS® is a trade mark registered in India (no. 6371251, Class 41). Ai4Testers™ is also used in relation to the Ai4Testers software. The document grants no license under any trade mark and no license to the software, its models, corpora, rule sets, lexicons or data.
Authors
- Ashwin Palaparthi (ORCID: https://orcid.org/0009-0000-0249-0312)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-19
- DOI
- https://doi.org/10.5281/zenodo.23861761
- Primary Topic
- Software Testing and Debugging Techniques
- Type
- article
- Field-Weighted Citation Impact
- 0.00