A Neuro-Symbolic Spatiotemporal Vision Transformer for Zero-Shot Safety Behavior Recognition

This study evaluates video-level behavior recognition on the seven-class Construction Meta Action (CMA) benchmark using a neuro-symbolic spatiotemporal Vision Transformer. An MViTv2-S backbone feeds a neural classification head and eight weakly supervised predicate variables, which are scored against fixed class prototypes; a temporal extension uses six log-domain operators over four ordered segments. On 540 official test videos, the fused model achieves 93.35% Macro-AP, compared with 93.23% for the neural branch. Zero-shot recognition is evaluated as leave-one-class-out transfer within CMA: the target class is excluded from task-specific head training and model selection, while Kinetics pretraining and target-class prototypes remain available. Mean held-out AP is 47.66% for the rule branch and 44.69% for the neural branch, without a statistically established difference. Log-domain computation maintains finite outputs and gradients under the reported stress conditions. The measured offline cost is 199.36 ms/video for single-clip file processing and approximately 1.17 s/video for four-segment extraction on an RTX 3080 Ti laptop GPU. The findings support inspection of prototype-based score construction and numerical behavior within the evaluated benchmark. Predicate semantics are not independently annotated, predictions are not attributed to individual workers, and cross-site generalization and live-stream performance have not been validated.

Authors

Institutions

Publication Details

Journal
Applied Sciences
Published
2026-11-12
DOI
https://doi.org/10.3390/app162211227
Primary Topic
Human Pose and Action Recognition
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A Neuro-Symbolic Spatiotemporal Vision Transformer for Zero-Shot Safety Behavior Recognition

Zhengwei Chang, Rui Song, Yi Zhang, Fangzhou Hu et al.
Applied Sciences
Human Pose and Action Recognition
article

A Neuro-Symbolic Spatiotemporal Vision Transformer for Zero-Shot Safety Behavior Recognition

Zhengwei Chang, Rui Song, Yi Zhang, Fangzhou Hu, Linghao Zhang, Ankai Zhang
article en

Abstract

This study evaluates video-level behavior recognition on the seven-class Construction Meta Action (CMA) benchmark using a neuro-symbolic spatiotemporal Vision Transformer. An MViTv2-S backbone feeds a neural classification head and eight weakly supervised predicate variables, which are scored against fixed class prototypes; a temporal extension uses six log-domain operators over four ordered segments. On 540 official test videos, the fused model achieves 93.35% Macro-AP, compared with 93.23% for the neural branch. Zero-shot recognition is evaluated as leave-one-class-out transfer within CMA: the target class is excluded from task-specific head training and model selection, while Kinetics pretraining and target-class prototypes remain available. Mean held-out AP is 47.66% for the rule branch and 44.69% for the neural branch, without a statistically established difference. Log-domain computation maintains finite outputs and gradients under the reported stress conditions. The measured offline cost is 199.36 ms/video for single-clip file processing and approximately 1.17 s/video for four-segment extraction on an RTX 3080 Ti laptop GPU. The findings support inspection of prototype-based score construction and numerical behavior within the evaluated benchmark. Predicate semantics are not independently annotated, predictions are not attributed to individual workers, and cross-site generalization and live-stream performance have not been validated.

Applied SciencesVol. 16(22)
China Electric Power Research Institute (CN), State Grid Sichuan Electric Power Corporation (China) (CN)
State Grid Corporation of China
Human Pose and Action Recognition
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon