Phys-Field: Physics-Enhanced Neural Surfaces for Sim(3) Estimation in Robotic Assembly Using a Single RGB Sensor

6-D pose estimation from a monocular RGB sensor is essential for robotic assembly. While deep learning approaches leverage priors from synthetic and labeled real-world data to estimate 6-D poses, their generalizability is often constrained by the limited scale and realism of training datasets. Moreover, scale ambiguity is an inherent issue when image datasets are captured without calibration or depth sensing. To address these limitations, we propose Phys-Field, a physics-aware neural surface framework for joint 6-D pose and scale estimation in robotic assembly without requiring 3-D models or depth sensing. We incorporate an isotropic scale factor into the 6-D pose estimation process, formulating the task as an iterative optimization problem within the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text {Sim(3)}$ </tex-math></inline-formula> group. The framework integrates multiple differentiable neural surfaces to estimate the corresponding <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text {Sim(3)}$ </tex-math></inline-formula> transformations by minimizing photometric differences between rendered and observed images through rendering inversion. To achieve conflict-free compositional rendering and efficient collision detection, we introduce deep convex decomposition (DCD). Concurrently, Phys-Field leverages dynamic simulations with physics parameters derived from large language models (LLMs) to evaluate the plausibility of the estimated <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text {Sim(3)}$ </tex-math></inline-formula> transformations, effectively resolving scale ambiguity and accelerating convergence. We validate our framework’s effectiveness in a real-world assembly task, with experimental results showing notable improvements in reconstruction fidelity, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text {Sim(3)}$ </tex-math></inline-formula> transformations estimation accuracy, and assembly success rate.