ChatGPT Responses in Hard Flaccid Syndrome: Exploratory Evaluation

Abstract Background Hard flaccid syndrome (HFS) is a rare and poorly described andrological condition characterized by a persistent semirigid penis in the flaccid state and frequently accompanied by erectile dysfunction, sensory disturbances, urinary symptoms, and psychosocial distress. Due to the absence of standardized diagnostic and therapeutic guidelines, patients increasingly seek health information from AI-based tools such as ChatGPT. However, the reliability of such tools in rare diseases remains uncertain. Objective This study aimed to perform an exploratory evaluation of the accuracy, readability, and short-term reproducibility of ChatGPT responses to patient-generated questions regarding HFS. Methods Sixteen clinically relevant questions spanning 4 domains (epidemiology, treatment, diagnostics, and symptoms and case presentation) were developed using authentic patient-derived queries and previously published AI evaluations in rare diseases. Questions were submitted to the ChatGPT web interface (OpenAI) in separate chat sessions and repeated after 48 hours to assess short-term reproducibility. Three urologists with expertise in male sexual health independently evaluated response accuracy using a validated 4-point scale. Readability was assessed using the Gunning Fog Index, whereas interrater reliability was evaluated using intraclass correlation coefficients (ICCs). Results Overall readability remained stable between sessions, with mean Gunning Fog Index scores of 15.19 (SD 1.68) at baseline and 14.59 (SD 1.66) after 48 hours ( P =.33), corresponding to an upper high school to early college reading level. No significant differences in readability were observed across domains at either time point (T1 [November 27, 2025]: P =.76; T2 [December 1, 2025]: P =.68). Accuracy differed across clinical domains. At T1, a total of 43.8% (7/16) of the responses were classified as correct and comprehensive, 50% (8/16) were classified as correct but inadequate, and 6.3% (1/16) were classified as having mixed correct and incorrect information; no responses were classified as completely incorrect. At T2, performance improved, with 62.5% (10/16) of responses classified as correct and comprehensive and 37.5% (6/16) classified as correct but inadequate. Interrater reliability demonstrated fair agreement at the individual reviewer level and moderate agreement when ratings were averaged (T1 single-measure ICC=0.36, 95% CI 0.06‐0.67; T1 average-measure ICC=0.63, 95% CI 0.17‐0.86; T2 single-measure ICC=0.33, 95% CI 0.04‐0.64; T2 average-measure ICC=0.59, 95% CI 0.10‐0.84). Short-term reproducibility was substantial, with 87.5% (14/16) of responses receiving identical final accuracy classifications between sessions. Conclusions ChatGPT demonstrated stable readability and high short-term consistency when addressing HFS-related questions. However, accuracy remained variable across clinical domains, with responses occasionally extending beyond the available evidence base in this poorly characterized condition. These findings suggest that AI tools may serve as accessible informational resources but should complement rather than replace expert clinical evaluation and evidence-based guidance, particularly in rare diseases for which clinical consensus remains limited.