A 70 Clip Trap Set for Measuring Invented Drug Names in Veterinary AI Scribes

AI scribes for veterinary medicine typically pair a speech recognition layer with a language model layer that drafts the clinical note. Either layer, or the two together, can add a medication to the record that nobody said. We built a blind, reproducible 70 clip test set, 45 ordinary dictations and 25 traps designed to bait a false drug name, written by an author with no access to the underlying matching or scoring code. We ran it once, on the same day, against a commercial veterinary AI scribe on a trial account with its default settings, and against our own scribe, Perch, with its out of the box lexicon. Eighty four drug mentions were dictated across the 45 dictations. We report raw counts, not rates: mentions found, mentions missed, false offers, and cases where one half of a dictated brand and generic pair was dropped. We describe every case where a product charted a drug that nobody said, including one from our own product's development history, and one case where the other product recovered a drug that our own product missed on this run. The full clip set, scoring script, and per clip ledger are published so anyone can rerun the test.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-08
DOI
https://doi.org/10.5281/zenodo.24057957
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

A 70 Clip Trap Set for Measuring Invented Drug Names in Veterinary AI Scribes

Mohammed Usama, Mustafa Nisar
2 links
Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence in Healthcare and Education
preprint

A 70 Clip Trap Set for Measuring Invented Drug Names in Veterinary AI Scribes

Mohammed Usama, Mustafa Nisar
preprint en

Abstract

AI scribes for veterinary medicine typically pair a speech recognition layer with a language model layer that drafts the clinical note. Either layer, or the two together, can add a medication to the record that nobody said. We built a blind, reproducible 70 clip test set, 45 ordinary dictations and 25 traps designed to bait a false drug name, written by an author with no access to the underlying matching or scoring code. We ran it once, on the same day, against a commercial veterinary AI scribe on a trial account with its default settings, and against our own scribe, Perch, with its out of the box lexicon. Eighty four drug mentions were dictated across the 45 dictations. We report raw counts, not rates: mentions found, mentions missed, false offers, and cases where one half of a dictated brand and generic pair was dropped. We describe every case where a product charted a drug that nobody said, including one from our own product's development history, and one case where the other product recovered a drug that our own product missed on this run. The full clip set, scoring script, and per clip ledger are published so anyone can rerun the test.

Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon