TaCOS: Task-Specific Camera Optimization with Simulation

The performance of perception tasks is heavily influ-enced by imaging systems. However, designing cameras with high task performance is costly, requiring extensive camera knowledge and experimentation with physical hard-ware. Additionally, cameras and perception tasks are mostly designed in isolation, whereas recent methods that jointly design cameras and tasks have shown improved performance. Therefore, we present a novel end-to-end optimization approach that co-designs cameras with spe-cific vision tasks. This method combines derivative-free and gradient-based optimizers to support both continuous and discrete camera parameters within manufacturing constraints. We leverage recent computer graphics techniques and physical camera characteristics to simulate the cam-eras in virtual environments, making the design process cost-effective. We validate our simulations against phys-ical cameras and provide a procedurally generated vir-tual environment. Our experiments demonstrate that our method designs cameras that outperform common off-the-shelf options, and more efficiently compared to the state-of-the-art approach, requiring only 2 minutes to design a camera on an example experiment compared with 67 min-utes for the competing method. Designed to support the development of cameras under manufacturing constraints, multiple cameras, and unconventional cameras, we be-lieve this approach can advance the fully automated de-sign of cameras. Code is available on our project page at https://roboticimaging.org/Projects/TaCOS/.