Digital twins based visual saliency smart home interaction design and virtual real validation through multimodal fusion

This study proposes a smart home interaction design method that integrates visual saliency modeling, multimodal interaction and digital twins (DTs) simulation, aiming at improving interaction efficiency, prediction accuracy and User Experience (UX) in multi-scene environment. This study discusses the construction of three kinds of DTs scenes of living room, bedroom and kitchen based on Unity, and introduces light field saliency prediction, sound field signal-to-noise ratio-confidence modeling, gesture trajectory generation and agent-based user behavior simulation to form a computable and verifiable virtual prototype. The system completed 450 virtual simulations, and combined with the real physical experiments of 20 users for cross-validation. The results show that the average comprehensive efficiency index (CEI) of users in the real experiment is 0.862, the average error rate is 5.97%, and the average UX score is 8.18. The average error of interaction time between virtual experiment and real experiment is 0.078, the error range is 0.061–0.089, the average difference of CEI is 0.018, and the error rate difference is about + 0.8%, which shows that the virtual prototype has stable prediction ability in the test scenario. In the design space optimization, Genetic Algorithm-Particle Swarm Optimization (GA-PSO) algorithm searches 300 groups of parameter combinations, which improves the CEI from 0.86 to 0.91, the task completion rate from 0.927 to 0.953, the error rate from 5.4% to 4.3%, and the average completion time from 13.2 s to 11.7 s. Cross-scene, cross-task and cross-modal experiments further show that multimodal fusion achieved better values than single-modal interaction under the tested metrics in efficiency, delay and UX. The combination of DTs, visual saliency modeling and multimodal fusion can provide quantifiable modeling, verification and optimization paths for smart home interaction design.