Optimizing Vision-Language Model for Road Crossing Intention Estimation

Identifying a pedestrian's intention to cross the road is crucial for autonomous driving, as it alerts the system to stop or slow down. However, determining crossing intention from video is challenging due to the need for extracting complex high-level semantics. This paper introduces ClipCross, a novel classification framework optimized to ex-tract high-level semantic features using the vision-language model CLIP for determining crossing intention. Existing CLIP-based methods perform poorly in this task, as CLIP's image and text encoders fail to capture the nuanced se-mantic distinctions between crossing and non-crossing in-tention images. Clip Cross addresses this by optimizing a set of CLIP text embeddings to extract high-level semantic features, which a multi-layer perceptron uses to distinguish between crossing and non-crossing intentions. Clip Cross achieves state-of-the-art performance on crossing intention estimation benchmark datasets: PIE, PSI, and lAAD.