DynaFusion: Dynamic Feature Fusion for Multimodal 3D Semantic Segmentation

Multimodal 3D semantic segmentation for autonomous driving benefits from fusing LiDAR and RGB, yet most architectures spread computation uniformly and overlook regions that are geometrically difficult or semantically ambiguous. We propose DynaFusion, a three-stage, difficulty-aware framework that concentrates compute where it matters. Stage-1 learns difficulty from voxel entropy to prioritize challenging points and learns saliency to reweight voxel features to emphasize informative voxels, then devoxelizes the voxel features and fuses them with point-wise image features to form initial embeddings. Stage-2 refines only the top-<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</i>% challenging points via dual deformable attention, performing learned sparse sampling on multi-view image features and on a voxel grid derived from the saliency-modulated features, followed by adaptive modality gating. Stage-3 refines only the top-<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">q</i>% semantically uncertain points, ranked by point-level entropy, using lightweight k-NN multi-head self-attention to aggregate local context. On the nuScenes <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">test</i> set, DynaFusion attains 82.4 mIoU. On the SemanticKITTI <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">test</i> set, it achieves 70.2 mIoU, surpassing strong LiDAR and LiDAR–camera baselines. These results demonstrate that difficulty-aware, point-selective refinement provides an effective alternative to uniformly applying the same fusion and refinement to all points in multimodal 3D segmentation. Moreover, under comparable computational budgets, DynaFusion offers a favorable mIoU–latency trade-off against recent LiDAR–camera fusion baselines on SemanticKITTI and nuScenes.