Fusion‐Guided and Distillation‐Optimized Framework for Freespace Detection in Off‐Road Environments

ABSTRACT In response to the challenges posed by the dynamic and diverse terrain of off‐road environments, the limited scale of data samples, and the insufficient generalization capability of models, this paper proposes an off‐road freespace detection framework called OFF‐LIP SAM. This framework employs a cascaded fusion approach, using dense point clouds as prompts to interactively guide the inference of the Segment Anything Model (SAM) vision large model. The framework first constructs a point cloud densification algorithm, calculating multi‐frame point cloud pose transformation matrices through a lidar inertial odometer and building a dense local point cloud map using a dynamic adaptive keyframe strategy. A dynamic keypoint sampling algorithm is then designed to construct feature point clouds, which are used as prior information to interactively generate regions of interest, enabling knowledge and data to jointly drive SAM model segmentation inference. Finally, this study proposes and implements a two‐stage knowledge distillation framework based on dynamic sampling prompts. This significantly improves the SAM inference speed while maintaining accuracy and generalization, making it applicable directly in resource‐constrained real‐vehicle environments. The experimental results demonstrate that the proposed OFF‐LIP SAM detection framework achieves competitive performance on the ORFD, RELLIS‐3D, and WildScenes data sets, outperforming current state‐of‐the‐art algorithms and strong baseline models. Real‐vehicle experiments conducted on the ORIN vehicle‐mounted computing platform validate the feasibility and effectiveness of the OFF‐LIP SAM detection framework.