Large Vison-Language Foundation Model in Baidu AIGC Image Advertising

Recent advances in generative artificial intelligence have revolutionized information retrieval and content generation, opening up new opportunities for the e-commerce industry. Alignment learning between small models and parallel corpora cannot meet current needs. The success of ChatGPT demonstrates that large models need to first establish a fundamental understanding, and then utilize high-quality corpora for generation. Having a large model foundation is indispensable. In this paper, we establish a fundamental 10B multimodal model foundation for multimodal generation tasks and propose a scene-based alignment learning approach called conditional sample supervised fine-tuning for downstream generation tasks. Meanwhile, diffusion models are known to be vulnerable to outliers in training data. To address this, we utilize an alternative diffusion loss function that preserves the high quality of generated data like the original squared L2 loss while being robust to outliers.In practical test sets, the multimodal foundation fully demonstrates its alignment and comprehension abilities for graphic and textual content. Additionally, conditional fine-tuning and the design of the loss function significantly enhance the quality of generated content. The quality rate of images has increased by 34.3 percentage points, and prompt control has improved by 19.8 percentage points. The application of our framework in Baidu Search Ads has led to significant revenue growth. For instance, ads with generated image creatives have achieved a 29% higher click-through rate (CTR), resulting in a daily consumption of 3 million yuan.