A Prototype of Relational Risk Telemetry for AI-Based Tools and Chatbots

Conversational systems based on large language models are being deployed as the first line of contact in customer service, yet their quality is still largely measured using process metrics that do not capture phenomena with potential impact on relational outcomes: lack of dialogue progress, failed error-recovery attempts, or silent customer departure without complaint. This paper presents a prototype of a conceptual and implementational framework for conversational risk telemetry, built on two components: an Operational Taxonomy of conversational states and events grounded in service recovery research, and LLM-as-a-Judge, an automated measurement instrument operating on structured rubrics with an evidence-span requirement. The prototype operates in a production environment: 1,069 scored Polish-language customer-service sessions (8,471 messages) under a single pipeline version, with 631 sessions forming the analytical corpus for trajectory-level statistics. In the pilot corpus, only 40% of sessions flagged for priority review close with negative sentiment, supporting the core premise that relational risk is procedural rather than tonal. A judge-model swap experiment on the full corpus quantifies the instrument's model sensitivity (weighted kappa = 0.70 on the priority band, rule-anchored fields effectively invariant at 99.7% agreement). Includes a pre-registered validation plan.