From Observation to Understanding: Front-Door Adjustments with Uncertainty Calibration for Enhancing Egocentric Reasoning in LVLMs

Recent progress in large vision-language models (LVLMs) has shown substantial potential across a broad spectrum of third-person tasks.However, adapting these LVLMs to egocentric scenarios remains challenging due to their third-person training bias.Existing methods that adapt LVLMs for first-person tasks often overlook critical agent-environment interactions, limiting their ability to perform egocentric reasoning.To address these challenges, we propose a novel zero-shot paradigm termed Front-Door Adjustments with Uncertainty Calibration (FRUIT) to enhance the egocentric reasoning abilities of LVLMs by simulating human causal reasoning.Specifically, the FRUIT operates in two stages: observation and understanding.Unlike conventional prompting techniques, we formalize egocentric reasoning using a structural causal model.Then, we ground interaction regions and expand them into hierarchical visual cues, augmented with corresponding captions, to form the initial observations.To reduce noise in these observations, we employ uncertainty calibration to filter out unreliable information.These refined observations as mediators are then incorporated into the prompt template, guiding the model to understand semantics from a firstperson perspective.Extensive experiments conducted on the EgoThink benchmark demonstrate that our FRUIT method consistently enhances the performance of existing LVLMs on six distinct tasks.Our code is available at https://github.com/Mrshenshen/FRUIT.