SLICE: Synthetic Caption-Trained Lightweight Image Captioner for Edge Devices

Image captioning has made significant steps forward in recent years. Existing state-of-the-art image captioning models are heavily parameterized and a long way from being deployable on edge devices. Providing image captioning inference on edge devices with current state-of-the-art models requires expensive server-side infrastructure and a stable network connection. In this paper, we propose Synthetic caption-trained Lightweight Image Captioning for Edge devices (SLICE). It provides state-of-the-art caption quality at a slice of the compute of existing large models. We achieve this by proposing a lightweight pipeline, redesigning the baseline architecture by making multiple modifications. The model is first pre-trained using our own generated synthetic captions and then fine-tuned by adopting a novel knowledge distillation approach to improve the caption quality. Our small-SLICE achieves 8.9x reduction in the number of model parameters, 7.9x reduction in peak memory consumption and 10.5x latency improvement, while surpassing model quality compared to the baseline model using a smaller input size achieving 138.9 CIDEr. Our large-SLICE achieves 147.3 CIDEr outperforming existing image captioning models with only 35.8M parameters on COCO Karpathy test split. On desktop CPU, our proposed small-SLICE achieves a low latency of 36ms.