LLM Assisted Attribute Generation for Image Datasets – A ChatGPT Case Study.

Large Language Models (LLMs) have advanced tremendously in recent years. Large-scale models such as GPT4, GEMINI, Llama, and Claude have opened new frontiers for what is possible for generative models. They’ve been utilized in a plethora of applications, including those that require tedious domain knowledge and subject matter expertise. One such application for the utilization of LLMs proposed in this study is the attribute generation process for both labeled and unlabeled datasets. Data labels, along with the image features/attributes, are crucial for the application of image datasets in causal reasoning tasks. Causal reasoning for image datasets cannot be accomplished without the proper data labels, which serve as the ground truth. However, manual data labeling and attribute/feature generation for image datasets is a tedious and sometimes unfeasible task. Therefore, we propose to use LLMs for automating the attribute generation and image labeling task for images. To address this, we investigate LLMs to determine whether they are capable of generating attributes and how correct these attributes are. Using two different datasets and a combination of prompt-tuning, we highlight mixed results for the ability of ChatGPT 4o to accurately classify the image label and generate attributes for the images based on the prompt, types of images, input methods, and whether fine-tuning is performed. For the AWA dataset, we highlight $100 \%$ accuracy for image classification and attribute generation when prompted with the options for the possible attributes. When the attribute options are not provided the results vary for both datasets. For the MRI scans, the accuracy of the image classification and attributes varied from $40 \%$ to $100 \%$ depending on whether the attribute options were provided and the level of granularity needed.