Simultaneous semantic segmentation and friction estimation using hierarchical latent variable models

This study redefines the problem of friction estimation as a visual perceptual learning task, advancing towards a joint per-pixel estimation of friction and surface classification by leveraging an onboard camera. By leveraging conditional variational autoencoder (CVAE), this research aims to develop a reliable system for friction estimation, ultimately enhancing vehicle safety and performance under varying environmental conditions.A new framework for unified friction estimation and segmentation is proposed, utilizing generalized ELBO with constrained optimization (GECO) to balance the trade-offs between friction estimation accuracy, segmentation accuracy, and the model’s robustness. This framework introduces an end-to-end approach that simultaneously addresses semantic segmentation and friction estimation within a unified system. The integrated methodology leverages the interrelations between surface identification and friction estimation for mutual benefit.Key challenges addressed in this project include ensuring cost-effectiveness and deployment across various vehicle types, managing environmental variability such as sun glare and rainy weather, and overcoming the lack of available data for friction estimation as a visual perceptual learning task. To tackle these challenges, a novel visual dataset for friction-related scene understanding was developed through data collection efforts, including participation in Volvo Trucks’ annual winter testing campaigns.The results suggest that the model trained on patch images of surfaces and later fine- tuned on the collected data achieved superior performance in both friction estimation and segmentation, with an intersection over union (IoU) score of 0.92, outperforming other models. The proposed model achieved per-pixel accuracies of 97% and 95% when identifying snow and ice respectively, and root mean squared error (RMSE) errors of 0.04–0.09 when estimating μ values achievable by a truck anti-lock braking system (ABS) on gravel, dry and wet asphalt, snow, and ice surfaces.