DREAM-CFA: joint learning of binary color filter array and demosaicing

Color filter arrays (CFA) are optical filters in digital cameras that facilitate capturing specific color channels by silicon sensors. Current commercial CFAs are hand-crafted patterns that are not necessarily optimal. We propose to learn a task-specific binary CFA from data utilizing a joint deep-learning architecture based on hard thresholding and learning-based demosaicing networks. Unlike existing learnable CFAs that learn a linear combination of color channels, the proposed method learns to select only one color channel at each pixel, resulting in CFAs that are practical and physically implementable to digital cameras. The proposed binary selection method adapts hard thresholding into neural networks via a straight-through estimator. A original demosaicing architecture then acquires a color image from the binary raw input masked by the learned CFA. Both color filter and demosaicing network are jointly learned over a training dataset to minimize the reconstruction loss. In this way, an optimal binary CFA is learned for the image reconstruction task for the training dataset. The proposed approach is tested with Kodak and BSDS500 datasets. Our results indicate that CFAs learned with the proposed approach provide a higher reconstruction performance than the hand-crafted filters such as Bayer or alternative learned CFAs. We provide an analysis of different demosaicing models, color configurations, CFA sizes, training dataset sizes, and loss functions to show the high performance of the proposed model.