Multimodal Large Model Image Data Augmentation Based on Cue Engineering

Data augmentation plays a vital role in tackling model overfitting in deep learning. Traditional methods of data augmentation typically utilize basic image manipulation or deep learning techniques, which face challenges such as insufficient information increase, semantic distortion, high training difficulty, and significant computational overhead. To overcome these issues, this paper proposes a multimodal large-model graphical approach based on cue engineering, called Randomized Strategy Mixing Data Augmentation (RSMDA), to achieve semantically invariant data augmentation. RSMDA leverages MiniGPT-5 to generate text from the original image data, followed by data cleaning to obtain the original text data. This text is then mixed using a random strategy. The randomized strategy mixing includes three common techniques: concurrent mixing, weighted mixing, and foreground-background cross mixing, allowing for manual intervention in data semantics. Finally, the blended text data undergoes a text-to-graph operation to produce the augmented image data. Expanding the CIFAR100-Subset database with RSMDA improves test accuracy by 3.5% to 20.3% compared to standard methods like Cutout and GIF-SD.