Fine-tuning image-to-text models on Liechtenstein tourist attractions

Abstract Adjusting pre-trained artificial intelligence models to domain-specific problems is essential for many business problems. But domain-specific data is often scarce and expensive to collect. Moreover, fine-tuning on small datasets is challenging, as it carries risks of overfitting and catastrophic forgetting. This paper systematically investigates the effectiveness of fine-tuning pre-trained image-to-text models for domain-specific applications, emphasizing how model performance scales with dataset size. We compare two state-of-the-art architectures, Generative Image-to-Text (GIT) and Florence-2, using small and large datasets of Liechtenstein tourism attractions. Our analysis reveals a nuanced relationship between model architecture and data efficiency. On the small dataset, measured by BLEU score, GIT outperformed Florence-2 (0.71 vs 0.03). However, with the larger dataset, Florence-2 surpassed GIT by 33–37%. Similarly, CIDEr scores improved dramatically from 0.00 to 0.97 for GIT and from 0.33 to 0.95 for Florence-2, underscoring the critical importance of data volume. Our results suggest that fine-tuned models are capable of generating contextually accurate captions, capturing architectural details, historical context, and geographical information of tourist attractions, as well as potentially benefiting other domains like cultural heritage preservation and education. Our methodology emphasizes computational efficiency, requiring less than 3 GB of GPU memory for both GIT and Florence-2, making these approaches accessible to organizations with limited resources. This research contributes both theoretical insights into model scaling properties and practical guidance on selecting appropriate architectures based on available data resources. The results demonstrate that while fine-tuning can enable reasonable performance even with limited domain-specific data, architecture selection should be informed by anticipated data availability. Furthermore, evaluating multiple models is highly recommended.