Evaluation of AI assisted problem solving based on ChatGPT accuracy and effectiveness in Adobe Photoshop user queries

This study aims to evaluate the content validity, completeness, and effectiveness of ChatGPT’s responses to common technical issues encountered by Adobe Photoshop (PS) users. Ten representative queries, categorized into five major usage domains, were posed to the ChatGPT interface powered by the GPT-4o model (accessed on March 10, 2025) under standardized conditions. The generated responses were evaluated by three independent experts using a multi-criteria framework incorporating 6-point (Accuracy), 3-point (Completeness), and 5-point (Effectiveness) Likert-type scales. Inter-rater reliability was examined using ICC (2,1) and percentage agreement metrics. In addition, the readability of the responses was analyzed using the Flesch-Kincaid Reading Ease and Grade Level formulas to determine linguistic accessibility. The evaluation revealed that ChatGPT’s responses exhibited consistently high accuracy scores (ranging from 5.0 to 6.0), while completeness and effectiveness scores showed greater variability across queries. The Flesch-Kincaid analysis indicated that most responses achieved a readability level between the 5th and 7th grades, suggesting suitability for novice to intermediate users. However, some responses lacked depth or contextual elaboration, particularly for more complex tasks. By focusing on Adobe Photoshop—a widely used creative software tool in higher education—this study addresses a gap in the existing literature, which has predominantly examined ChatGPT in domains such as programming, clinical decision-making, and theoretical content. Overall, the findings highlight ChatGPT’s potential to provide technically accurate and user-friendly support in creative software environments while identifying areas for improvement in content richness and domain-specific adaptability. The study contributes to the growing body of research on AI-assisted problem solving in specialized user contexts and offers implications for instructional design and digital learning practices in higher education.