Re-Purposing Segment Anything For Skeleton Action Localization

The Segment Anything Model (SAM) establishes a foundational model for prompt-based image segmentation. SAM interactively segments areas of interest using simple point, box, and/or text prompts. However, applying SAM to non-image segmentation tasks remains an open challenge. For instance, localization of actions in an untrimmed skeleton sequence presents its own diverse set of challenges due to small visual and semantic differences around action boundaries. This work proposes a prompt-based action localization method, referred to as PAL-SAM, by repurposing SAM to locate actions within untrimmed skeleton sequences using frame prompts. To tackle the challenging transitional boundaries between actions, a boundary-aware prompt decoder is introduced to improve localization accuracy. Experiments on the public dataset PKU-MMD confirm that PAL-SAM is able to localize actions with frame prompts and shows promise for extension to other types of prompts.