GenAI applications of vision-language models for semiconductor defect classification

Visual Language Models (VLMs) combine advanced image/video recognition with the dialog capabilities of Large Language Models (LLMs), bringing AI applications closer to human-like intelligence. In the field of semiconductor imaging, this is an industry-first case study demonstrating the integration of VLMs for detection and classification—paving the way for entirely new solutions in semiconductor imaging. Since the advent of deep learning, CNN-based models have helped the semiconductor industry achieve success in anomaly classification, including optical and e-beam inspection, defect distribution maps, and even lithographic quality assessment. However, building a robust model typically requires thousands of images to reach high accuracy, and retraining is necessary whenever new products, processes, or defect types emerge—especially challenging during the initial stages of data collection. In contrast, our proposed method fine-tunes a VLM to enhance its understanding of semiconductor images (including defects), enabling classification tasks with fewer training samples—an approach aligned with the concept of few-shot learning. Additionally, we have incorporated a self-supervised Vision Transformer (ViT) model—hereafter referred to as the Vision Foundation Model (VFM)—to demonstrate its applications in defect classification via self-supervised learning, while requiring minimal domain-specific engineering.. This ensures our AI solutions can effectively address the diverse characteristics inherent to semiconductor imaging. Moreover, large-scale fabs can produce over 1,000 samples per day for Transmission Electron Microscopy (TEM) analysis, with each sample generating more than five images. Training engineers to manage this workload is both time-consuming and costly, whereas a fine-tuned VLM can assist in both image interpretation and analysis. Our ultimate goal is to integrate results from tens of thousands of images into concise summaries. Furthermore, VLMs can aid in understanding Statistical Process Control (SPC) charts by flagging potential issues before they exceed control or specification limits, or by comparing images to a “golden” reference to pinpoint anomalies. By delivering image insights in a conversational format, VLMs are poised to drive advancements in root-cause analysis and shape the future of agentic AI. In this presentation, we will highlight several downstream tasks for semiconductor imaging where VLMs have been successfully applied. We will share industry-first examples and demonstrate how to fine-tune a VLM for real-world semiconductor applications.