TrustToken: A Framework for Evaluating Tokenizer Security and Robustness in NLP Pipelines

The growing reliance on tokenizers in NLP systems calls for robust security measures. TrustToken, a framework for evaluating tokenizer trustworthiness across eight key metrics, including SQL Injection Risk Score (SIRS) and XSS Vulnerability Score (XVS), is introduced in this study as a proposed measure. A custom dataset designed to expose vulnerabilities and test resilience against malformed inputs, perturbations, and boundary conditions assesses GPT-2, BERT, T5, and RoBERTa security posture. While T5 achieved the highest Trustworthiness Score (TWS) of 0.77, all other models failed SQL Injection and XSS tests. Despite firm handling of special characters and privacy safeguards, weaknesses in processing malformed inputs and boundary conditions were evident. Namely with GPT-2, BERT, and RoBERTa. These findings underscore the security risks of deploying tokenizers in real-world applications and emphasize the need for input sanitization and adversarial training. The TrustToken framework offers a systematic approach to evaluating and strengthening tokenizers, ensuring safer NLP deployments.