Machine Learning-Based Detection of Jailbreak and Prompt Injection Attacks in Large Language Models

Large Language Models (LLMs) are now built into many everyday applications, which exposes them to jailbreak and prompt injection attacks that try to override safety rules or hijack the instructions a model follows. This paper presents a machine learning framework that screens prompts and flags likely malicious ones before they reach the model. We compare Logistic Regression, a Linear Support Vector Machine (SVM) and a fine-tuned DistilBERT classifier on a combined dataset of 24,530 prompts drawn from GuardrailsAI and Neuralchemy, prepared through label normalization, duplicate checking and stratified splitting. After the decision threshold was tuned on validation data, DistilBERT reached 95.56% accuracy, 93.74% precision, 88.77% recall and a 91.18% F1-score on the held-out test set. On a separate collection of 300 JailbreakBench-associated prompts, however, accuracy fell to about 82% and 55 of 200 attack prompts went undetected, which shows how much harder diverse adversarial inputs are than in-distribution data. The results suggest that transformer-based screening can serve as a useful additional security layer for LLM applications, provided it is validated on external benchmarks and combined with other defences.

Authors

Institutions

Publication Details

Journal
International Journal of Advanced Research in Science Communication and Technology
Published
2026-11-19
DOI
https://doi.org/10.48175/ijarsct-40870
Primary Topic
Adversarial Robustness in Machine Learning
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Machine Learning-Based Detection of Jailbreak and Prompt Injection Attacks in Large Language Models

Jatin Kushwaha, Diya Sadhu, Vijaya Patil, Anandita Kashyap Soham Sonawane
International Journal of Advanced Research in Science Communication and Technology
Adversarial Robustness in Machine Learning
article

Machine Learning-Based Detection of Jailbreak and Prompt Injection Attacks in Large Language Models

Jatin Kushwaha, Diya Sadhu, Vijaya Patil, Anandita Kashyap Soham Sonawane
article en

Abstract

Large Language Models (LLMs) are now built into many everyday applications, which exposes them to jailbreak and prompt injection attacks that try to override safety rules or hijack the instructions a model follows. This paper presents a machine learning framework that screens prompts and flags likely malicious ones before they reach the model. We compare Logistic Regression, a Linear Support Vector Machine (SVM) and a fine-tuned DistilBERT classifier on a combined dataset of 24,530 prompts drawn from GuardrailsAI and Neuralchemy, prepared through label normalization, duplicate checking and stratified splitting. After the decision threshold was tuned on validation data, DistilBERT reached 95.56% accuracy, 93.74% precision, 88.77% recall and a 91.18% F1-score on the held-out test set. On a separate collection of 300 JailbreakBench-associated prompts, however, accuracy fell to about 82% and 55 of 200 attack prompts went undetected, which shows how much harder diverse adversarial inputs are than in-distribution data. The results suggest that transformer-based screening can serve as a useful additional security layer for LLM applications, provided it is validated on external benchmarks and combined with other defences.

International Journal of Advanced Research in Science Communication and Technology
MIT Art, Design and Technology University (IN)
Adversarial Robustness in Machine Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon