Hybrid Surveillance for Abnormal Behaviour Detection with TST and Semantic AI

The "Hybrid Surveillance for Abnormal Behaviour Detection with TST and Semantic AI" proposes an advanced framework designed to enhance college safety by detecting abnormal behaviours in surveillance videos through a synergistic integration of deep learning and semantic analysis. The system employs a Temporal Segment Transformers (TST) model, meticulously trained on the UCF-Crime dataset comprising 111,308 images organized into 37,098 temporal segment groups, to analyse video frames grouped into three distinct segments (early, middle, late). Utilizing a pre-trained Vision Transformer (ViT) fine-tuned for binary classification, the TST achieves a Top-1 accuracy of 83.57% and a Top-5 accuracy of 97.16% in distinguishing normal activities from abnormal ones such as fighting or vandalism. This initial output is then processed by the Video Behaviour Analyzer, a Flask-based web application that segments the video into 10-second chunks with 2-second overlaps and leverages the Google Gemini API for detailed semantic categorization, producing precise behaviour labels like "Explosion" with confidence scores (e.g., 0.95) based on contextual analysis. By combining the temporal coherence of TST with the semantic precision of the AI-driven analyser, this hybrid system delivers a robust, scalable, and user-friendly solution for real-time campus monitoring. The intuitive web interface provides immediate feedback, displaying aggregated results and chunk-level breakdowns, making it accessible for practical deployment in educational institutions. While the system demonstrates high efficacy, challenges such as the 15-second processing delay per chunk suggest opportunities for future optimization, alongside potential expansions of the dataset to encompass a broader range of behaviours, further strengthening its applicability in diverse surveillance scenarios.