A generalizable pipeline for specialty-curated postmarket device surveillance: a reference implementation in neurosurgery

Spontaneous-reporting medical-device databases, including the US Food and Drug Administration (FDA) Manufacturer and User Facility Device Experience (MAUDE) database, store most of their content as unstructured free text. A generalizable methodology for converting these narratives into structured, searchable data is needed for specialty-level surveillance. We developed a 4-layer AI pipeline and applied it to the complete neurosurgical-device subset of the FDA MAUDE database from inception through March 31, 2026. Neurosurgical device product codes were identified from the complete FDA Product Classification list using a dual large language model (LLM) classifier with human adjudication. Each report was processed by: (1) dual-LLM specialty filtering of FDA product codes; (2) schema-constrained narrative labeling using a 15-class failure-mode taxonomy and an 8-class outcome taxonomy, with identifier-based input-output joining; (3) asymmetric semantic embedding into a 768-dimensional vector space indexed with hierarchical navigable small-world (HNSW) graphs and 8-bit scalar quantization; and (4) precomputed device-, manufacturer-, and category-level aggregate roll-up. Outcomes included labeling coverage; inter-LLM agreement on the specialty filter (Cohen κ); semantic-retrieval quality (Jaccard@10 against exhaustive nearest-neighbor search; top-1 match rate); and population-level label-to-record alignment (content-keyword hit rate; random-baseline expectation, 6.7% under the 15-class taxonomy). The neurosurgical corpus comprised 2,086,176 reports filed against 10,102 distinct devices from 641 manufacturers across 9 categories. The specialty filter evaluated 7,058 candidate FDA product codes and, after adjudication, yielded 527 neurosurgical codes. Against the adjudicated reference, the primary classifier (gemini-2.5-flash) had a sensitivity of 0.42, a specificity of 0.996, and a positive predictive value of 0.89, and the second-opinion classifier (claude-sonnet-4-5) had a sensitivity of 0.39, a specificity of 1.00, and a positive predictive value of 1.00; the 2 models agreed on 6,947 of 7,058 codes (Cohen κ = 0.75), and human adjudication with a recall-validation pass resolved discordant codes and recovered false negatives. The pipeline produced complete, schema-valid labels for 2,072,457 of 2,072,722 narratives (99.99% labeling coverage). Semantic-retrieval Jaccard@10 against exhaustive search was 0.83 (top-1 match rate, 0.81). Population-level content-keyword hit rates ranged from 32.9% to 91.5% across keyword classes, against the 6.7% rate expected under random assignment. Against an adjudicated reference standard from 2 clinically qualified reviewers, failure-mode labels reached 67% accuracy and outcome labels 58% (both far above majority-class baselines; inter-reviewer κ 0.90 and 0.85), and clinician-rated semantic retrieval reached precision@10 of 0.85. A 4-layer AI pipeline, designed to be generalizable, structured the free-text content of a national device adverse-event database on a proof-of-concept specialty subset. The architecture may transfer to other MAUDE specialty subsets and to analogous spontaneous-reporting databases.