A Multi-Mode Deep Learning Framework for Suspicious Behavior Detection in Controlled Environments

Surveillance in controlled environments such as military bases and other restricted facilities requires timely detection of suspicious activities. This paper presents a real-time multi-mode deep learning framework that integrates YOLOv8-based person detection, Deep SORT multi-object tracking, spatio-temporal feature extraction, and behavior recognition through rule-based, supervised, and self-learning operating modes. A six-mode ablation study compares hybrid and no-fallback variants under a shared deployment pipeline. Tracking evaluation on the PETS09 surveillance dataset confirms stable person trajectories suitable for downstream analysis, and a two-stage sensitivity study identifies operational thresholds for loitering duration, radius, and running speed. Under independent manual ground-truth validation on a controlled deployment video, the self-learning hybrid mode achieved the highest accuracy and macro F1 among the six evaluated modes, while supervised-synthetic and rule-based modes remain competitive. Removing the rule-based fallback causes both LSTM-only ablations to degrade toward near-random behavior, demonstrating that fallback fusion is functionally essential. Self-learning achieves this performance without any manually labeled deployment data, suggesting potential practical applicability for controlled surveillance environments, pending further validation. Preliminary cross-scene evaluation on a second deployment video shows that all hybrid modes maintain high agreement with a rule-based reference and reproduce the same no-fallback degradation pattern across scenes. On the manually annotated deployment subset, the self-learning hybrid mode attains an accuracy of 0.899 and a macro F1 of 0.713, while both no-fallback ablations collapse to near-random accuracy of approximately 0.50, a 38-40 percentage-point accuracy drop across the two ablations relative to their fallback-enabled counterparts, indicating that rule-based fallback fusion is necessary for maintaining above-chance performance in this framework.

Authors

Publication Details

Journal
Cureus Journal of Computer Science.
Published
2026-11-19
DOI
https://doi.org/10.7759/s44389-026-00381-8
Primary Topic
Video Surveillance and Tracking Methods
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A Multi-Mode Deep Learning Framework for Suspicious Behavior Detection in Controlled Environments

M. Akhtaruzzaman, Rabbi Mushad
Cureus Journal of Computer Science.
Video Surveillance and Tracking Methods
article

A Multi-Mode Deep Learning Framework for Suspicious Behavior Detection in Controlled Environments

M. Akhtaruzzaman, Rabbi Mushad
article en

Abstract

Surveillance in controlled environments such as military bases and other restricted facilities requires timely detection of suspicious activities. This paper presents a real-time multi-mode deep learning framework that integrates YOLOv8-based person detection, Deep SORT multi-object tracking, spatio-temporal feature extraction, and behavior recognition through rule-based, supervised, and self-learning operating modes. A six-mode ablation study compares hybrid and no-fallback variants under a shared deployment pipeline. Tracking evaluation on the PETS09 surveillance dataset confirms stable person trajectories suitable for downstream analysis, and a two-stage sensitivity study identifies operational thresholds for loitering duration, radius, and running speed. Under independent manual ground-truth validation on a controlled deployment video, the self-learning hybrid mode achieved the highest accuracy and macro F1 among the six evaluated modes, while supervised-synthetic and rule-based modes remain competitive. Removing the rule-based fallback causes both LSTM-only ablations to degrade toward near-random behavior, demonstrating that fallback fusion is functionally essential. Self-learning achieves this performance without any manually labeled deployment data, suggesting potential practical applicability for controlled surveillance environments, pending further validation. Preliminary cross-scene evaluation on a second deployment video shows that all hybrid modes maintain high agreement with a rule-based reference and reproduce the same no-fallback degradation pattern across scenes. On the manually annotated deployment subset, the self-learning hybrid mode attains an accuracy of 0.899 and a macro F1 of 0.713, while both no-fallback ablations collapse to near-random accuracy of approximately 0.50, a 38-40 percentage-point accuracy drop across the two ablations relative to their fallback-enabled counterparts, indicating that rule-based fallback fusion is necessary for maintaining above-chance performance in this framework.

Cureus Journal of Computer Science.
Video Surveillance and Tracking Methods
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon