NP-003 Abstract withdrawn

<h3>Background</h3> An initial survey on the performance of ChatGPT 3.5 in drug information revealed its performance and risks. To investigate the dynamics in the development of large language models, the analysis was repeated with ChatGPT 4 in January 2024. <h3>Research Question</h3> How has the performance of ChatGPT 4 changed in terms of response quality and safety in drug information? The initial sample of questions was answered again by ChatGPT 4. The quality of the answers was assessed in a structured consensus by hospital pharmacists based on the categories of content, patient management and risk and compared with the answers from ChatGPT 3.5. <h3>Results</h3> The content quality of ChatGPT 4 improved compared to its predecessor. Out of the 50 questions, ChatGPT 4 answered more questions correctly 38% (n=19) compared to 26% (n=13) and significantly fewer incorrectly (12%, n=6 compared to 38%, n=19), but these were less often feasible. Management was feasible for 52% (n=26) of the questions and impossible for 16% (n=8). For ChatGPT 3.5, the corresponding values were 72% (n=36) and 14% (n=7). We found 10% (n=5) of ChatGPT 4 responses resulted in a high patient risk and 52% (n=26) in no patient risk compared to 14% (n=7) and 46% (n=23) for ChatGPT 3.5. Overall, the number of responses that were correct, feasible and without risk improved from the initial 26% (n=13) to 36% (n=18) for ChatGPT 4. Compared to ChatGPT 3.5, ChatGPT 4 responses were longer on average (217.6 to 112.6 words) and the number of referenced responses increased (8% to 0%). <h3>Discussion</h3> ChatGPT 4 showed improved performance compared to the previous version, particularly in terms of content and risk. The increased word count does not necessarily correlate with improved practicability. Nevertheless, an assessment by specialist personnel is essential for safe advice.