WebXR-based mobile point cloud rendering and semantic segmentation using depth data and voxel fusion

A novel framework was developed for interactive, on-device visualization and user-triggered semantic segmentation of point clouds on mobile devices by integrating WebXR, THREE.js, and WebGL. Despite advances in mobile hardware, these tasks remain computationally intensive. The proposed framework leverages the WebXR API to acquire camera pose and depth data for efficient point cloud generation using BufferGeometry, followed by two-dimensional image recognition. A dynamic voxel fusion buffer was developed to construct voxel-based point clouds by aggregating semantic segmentation results from multiple frames. Segmentation results are further refined using a buffer voxel-counting algorithm and rendered in real time within a web browser using ShaderMaterial. Shaders are used to visualize public indoor datasets and convert them into depth maps, creating data compliant with WebXR constraints. The framework was trained and evaluated using the Stanford 3D Indoor Scene Dataset (S3DIS), and a lightweight 2D semantic segmentation model—trained on annotated real-world indoor scenes—achieved 89% accuracy and a mean Intersection over Union of 76%. The system was deployed and demonstrated in indoor scenes on a WebXR-capable mobile device. Under the tested configuration, the framework supports interactive visualization and user-triggered semantic segmentation; comprehensive latency, resource-use, cross-device, and environmental-robustness benchmarks remain subjects for future work.