AI Security Risk Analyzer: A Bilingual, Evidence-Calibrated Transformer System for Document Security Risk Detection

Commercial data leakage prevention (DLP) tools are expensive, typically featuring a severe bias towards English, and neglecting Indonesian as a language used in mixed-indicative documents containing the sixteen-digit KTP/NIK number of the Republic of Indonesia’s official citizen identification card. This paper proposes AI Security Risk Analyzer, a web application for identifying fifteen classes of security-sensitive information in either English or Indonesian from pasted text or documents in PDF, DOCX, PPTX, XLSX, CSV, TXT, and image formats (in OCR text). A fine-tuned XLM-RoBERTa encoder with evidence-based calibration to the output scores (bounding it to findings by rule-based pattern-matching) improves upon the two first-generation DistilBERT pipelines’ false discovery of sensitive information in benign structured documents and calibration issues in context-sensitive content (e.g., the value of a cell adjacent to special column headers). The output is expressed in terms of generic ISO/IEC 27005, NIST 800-30 Rev. 1, and FIPS 199 risk classification terminology with necessary caveats. The paper evaluates the system’s performance, demonstrating 86.3% of validation set accuracy and 80.8% macro F1 for fifteen-class classification, complete separation of expected-safe and expected-risk samples in the calibration set ten-case benchmark at the system’s own risk levels, and successful completion of all functional and calibration tests. The system is free to use, provided on free-tier cloud services, and designed as an adjunct to, rather than a replacement for, human security analysts.