Decomposing Video Activity Labeling into Human-Inspectable Steps: A Gray-Box Workflow via Narration-Derived Intermediate Outputs

Video activity labeling divides a video into activity segments and assigns a category to each. Manual annotation requires sequential review, while end-to-end AI models may require retraining when the coding task or taxonomy changes. We propose Narration-Mediated Activity Annotation (NMAA), a gray-box workflow that organizes transcript sentences into activity units and compares the resulting annotations with a human reference to evaluate where one activity ends and another begins (B), how activities are classified (T), and which transcript words support each classification (E). Researchers revise the coding protocol based on these results and observed disagreements, reevaluate it, and retain a revision only when its stated purpose is achieved without a confirmed loss in any of the three measures. We evaluated NMAA on 35 Roblox Restaurant Tycoon videos (13.8 h, 16,707 sentences) using a 1200-sentence human reference. The largest stepwise change came from assigning categories sentence by sentence and merging adjacent same-category sentences afterwards, which produced the largest increase in evidence agreement and brought mean activity-unit length close to the human reference. On the held-out test set, the final protocol improved B, T, and E by 0.100, 0.163, and 0.202, respectively, relative to the baseline. The same revision reduced activity-unit length for all three large language models from different providers. When the full corpus was analyzed with the final protocol, Facility and Interior Management accounted for a larger share of the creators’ narrated management time than of textbook topic coverage, whereas Marketing and Promotion accounted for a smaller share.

Authors

Institutions

Publication Details

Journal
Applied Sciences
Published
2026-11-17
DOI
https://doi.org/10.3390/app162211363
Primary Topic
Video Analysis and Summarization
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Decomposing Video Activity Labeling into Human-Inspectable Steps: A Gray-Box Workflow via Narration-Derived Intermediate Outputs

Mi-Na Jo, Hongryul Ahn, Seunghyun Kim
1 links
Applied Sciences
Video Analysis and Summarization
article

Decomposing Video Activity Labeling into Human-Inspectable Steps: A Gray-Box Workflow via Narration-Derived Intermediate Outputs

Mi-Na Jo, Hongryul Ahn, Seunghyun Kim
article en

Abstract

Video activity labeling divides a video into activity segments and assigns a category to each. Manual annotation requires sequential review, while end-to-end AI models may require retraining when the coding task or taxonomy changes. We propose Narration-Mediated Activity Annotation (NMAA), a gray-box workflow that organizes transcript sentences into activity units and compares the resulting annotations with a human reference to evaluate where one activity ends and another begins (B), how activities are classified (T), and which transcript words support each classification (E). Researchers revise the coding protocol based on these results and observed disagreements, reevaluate it, and retain a revision only when its stated purpose is achieved without a confirmed loss in any of the three measures. We evaluated NMAA on 35 Roblox Restaurant Tycoon videos (13.8 h, 16,707 sentences) using a 1200-sentence human reference. The largest stepwise change came from assigning categories sentence by sentence and merging adjacent same-category sentences afterwards, which produced the largest increase in evidence agreement and brought mean activity-unit length close to the human reference. On the held-out test set, the final protocol improved B, T, and E by 0.100, 0.163, and 0.202, respectively, relative to the baseline. The same revision reduced activity-unit length for all three large language models from different providers. When the full corpus was analyzed with the final protocol, Facility and Interior Management accounted for a larger share of the creators’ narrated management time than of textbook topic coverage, whereas Marketing and Promotion accounted for a smaller share.

Applied SciencesVol. 16(22)
University of Suwon (KR), Suwon Science College (KR)
National Research Foundation of Korea
Video Analysis and Summarization
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon