Methodological considerations and optimization directions for bibliometric research

Dear Editor, Large language models (LLMs) are driving digital transformation in healthcare as a generative AI tool with excellent comprehension and generation capabilities. In different scenarios of clinical practice, LLMs are expected to improve the efficiency of healthcare services due to their excellent text comprehension and generation capabilities in a variety of critical tasks, such as disease diagnosis and differential diagnosis scenarios, medical imaging report writing and interpretation scenarios, individualized treatment recommendation scenarios, patient health education scenarios, and the construction of a medical intelligent consultation system, which will support clinical decision-making and alleviate the clinical information load[1,2]. However, in the face of complex and variable real clinical scenarios, the accuracy, robustness, and reliability of LLMs in clinical applications at this stage still need to be further improved. Generative hallucinations may directly threaten the safety of diagnosis and treatment. Data privacy and security still limit the clinical application of LLMs. The ethical and legal responsibility of AI-involved medical decision-making remains unclear[1-3]. Therefore, there is an urgent need for more systematic research in this field to comprehensively evaluate and optimize the accuracy and safety of LLMs in real clinical scenarios, as well as to establish interpretable and auditable application specifications. We sincerely appreciate Tang et al’s constructive comments on our recent study, “Current concerns and future directions of LLM ChatGPT in medicine: a machine-learning-driven global-scale bibliometric analysis”[2,4]. Their perspectives helped us to further clarify the boundaries of our study, strengthen methodological rigor, and provide directions for subsequent improvements, thereby enhancing the credibility and reproducibility of our findings. Therefore, we take this valuable opportunity to further clarify the methodological considerations in our study and point out possible directions for future improvement. This study has been reported in line with the TITAN criteria[5]. Regarding the range of data sources, given the importance of research quality, our previous study mainly relied on the Web of Science Core Collection (WOSCC) and excluded non-English literature[6-10], but this may also have resulted in a narrow search. This is a balance between coverage and standardization that is often involved in bibliometric studies. The main reason for choosing the WOSCC for our study is that its index fields and citation relationships are more standardized, which facilitates cross-disciplinary comparisons, collaborative network construction, and stable extraction of metrics. Meanwhile, in order to reduce the single database bias and to check the stability of the algorithm results, we explicitly used Scopus as the external validation data source in the Methods section for the unsupervised clustering and Walktrap analysis. Regarding the inclusion and exclusion of non-English and non-peer-reviewed literature, we explained the rationale in the inclusion and exclusion criteria: language communication barriers and peer-review quality assurance. It is also explained that non-peer-reviewed literature was included in the analysis at the same time in order to obtain the outer layer of information when unsupervised clustering was performed to reduce the loss of information due to low-frequency label filtering and to test the reliability of the results in conjunction with external validation by Scopus. However, it is also true that subsequent studies can further incorporate PubMed, regional databases, and multilingual literature, and conduct relevant analyses under strict de-emphasis, synonym normalization, and bias control in order to enhance the global representativeness of the study and the completeness of the conclusions. Previous correspondence has suggested that using the median as a threshold for high development may not adequately reflect differences in study size across specialties. However, we add the original intent of our study design that to facilitate cross-specialty controls, we used a dichotomous definition that is transparent, reproducible, and maintains a relative balance of categories. As presented in the methodology section, we treated each assessment instance as an independent sample and defined high development as the number of application cases exceeding the median. At the same time, we adopt several robustness measures, such as Bonferroni correction, Hosmer-Lemeshow fit test, VIF covariance diagnostics, and Firth correction for small-sample disciplines to enhance the stability and interpretability of the study. Meanwhile, we also agree with the improvement strategies proposed in the correspondence. Future studies could conduct sensitivity analysis or introduce indicators reflecting the body of disciplines to portray the heterogeneity of disciplinary development in a finer way and further enhance the robustness of the conclusions. With regard to internal indicators of clustering, the correspondence points to the introduction of internal validity indicators such as profile coefficients or modularity, which would allow for the assessment of intraclass cohesion and interclass segregation. We are very grateful for this suggestion. The reason why we emphasize the reproducibility rate in our study is that we want to validate the reproducibility of the topic labels with external databases, so as to demonstrate the stability and reliability across databases. We also provide a rigorous description of the keyword network construction and cluster definition process in our methodology, including de-duplication, lowercase normalization, low-frequency filtering, link-strength normalization, and naming clusters based on weighted composite scores of total link-strength and occurrence frequency with semantic integration. We also recognize that supplementing internal metrics such as modularity for network structure adaptation in subsequent work will make the methodological presentation more complete and help to further support the credibility of topic identification from a structural quality perspective. Regarding synonymous merging of keywords, this manuscript updates our previous data based on the authors’ suggestions (Table 1). In our previous study, preprocessing consisted of de-duplicating words with uniform lowercase but did not provide adequate synonym merging for hyphen or space differences. We strongly agree that implementing stricter synonym merging and format normalization, such as hyphens or spaces, would further improve the consistency and interpretability of the results. Table 1 - Quantitative information about the corresponding labels in the six clusters Cluster Label Total link strength Occurrence Average publication year Average citation Cluster 1: The Risks, Challenges, and Policies of ChatGPT in Clinical Practice Technology 88 15 2023.0000 2.2667 Risk 72 12 2023.0000 3.1667 Healthcare 69 10 2023.0000 4.9000 Conversational agents 53 5 2023.0000 7.2000 Challenges 44 9 2023.2222 3.7778 Clinical practice 44 6 2023.0000 3.5000 Policy 43 6 2023.0000 2.8333 Acceptance 42 5 2023.0000 3.6000 Generative pretrained transformer 41 8 2023.0000 8.3750 Model 40 5 2023.0000 3.4000 Trust 39 6 2023.0000 5.8333 Self-diagnosis 34 3 2023.0000 5.6667 Barriers 30 3 2023.0000 6.6667 Knowledge 28 8 2023.0000 2.7500 Attitude 28 3 2023.0000 5.0000 Evaluation 26 4 2023.0000 0.2500 Cluster 2: The Applications of ChatGPT in Oncology Patient Management and Decision-Making Natural language processing 292 62 2023.0161 4.5484 Machine learning 237 53 2023.0566 11.2264 Deep learning 71 16 2023.1250 5.6875 Cancer 59 15 2023.0667 1.7333 Decision-making 50 6 2023.0000 3.3333 Oncology 45 12 2023.0000 1.5833 Patient 42 9 2023.0000 6.3333 Management 27 11 2023.0000 1.4545 Models 25 5 2023.2000 0.4000 ChatGPT 24 16 2023.0000 2.4375 Classification 24 6 2023.1667 1.6667 Disease 23 5 2023.0000 0.4000 Telemedicine 22 5 2023.0000 13.0000 Prediction 21 6 2023.1667 1.0000 Radiotherapy 19 5 2023.0000 3.0000 Cluster 3: The Applications of ChatGPT in Patient Education Artificial-intelligence 238 60 2023.0667 9.1333 Patient education 103 25 2023.0400 1.6000 Internet 72 16 2023.0000 1.6250 Quality 63 9 2023.0000 2.0000 Communication 50 9 2023.0000 3.4444 Generative artificial intelligence 46 15 2023.0000 0.8000 Readability 40 11 2023.0000 1.3636 Health literacy 38 8 2023.0000 5.5000 Information 35 8 2023.1250 5.8750 Google 33 7 2023.0000 2.4286 Algorithm 33 3 2023.3333 4.3333 Patient safety 30 5 2023.0000 5.6000 Care 26 7 2023.0000 27.5714 Urology 25 7 2023.1429 3.0000 Cluster 4: The Accuracy and Safety of ChatGPT in Health Information Recommendation Chatbot 459 96 2023.0321 8.5022 Large language model 102 13 2023.0000 2.8462 Accuracy 91 19 2023.1053 4.1053 Ophthalmology 85 17 2023.0000 8.2941 Recommendation 70 7 2023.0000 2.6667 Health information 53 8 2023.0000 1.3750 Impact 49 9 2023.0000 22.6667 Safety 49 6 2023.1667 5.5000 Therapy 48 8 2023.0000 1.0000 Dermatology 36 6 2023.0000 1.1667 Guidelines 35 6 2023.0000 4.6667 Reliability 30 5 2023.0000 1.4000 Medical information 28 3 2023.0000 7.6667 Cluster 5: The Applications of ChatGPT for Patient Stratification and Decision Support ChatGPT 1666 562 2023.0302 4.2651 Large language model 254 67 2023.0746 2.4179 Generative AI 72 17 2023.0588 2.2353 Artificial intelligence in medicine 44 12 2023.0000 20.2500 AI chatbot 28 8 2023.0000 3.3750 Triage 27 7 2023.1429 1.4286 Clinical decision support 41 8 2023.0000 5.1667 Decision-making 15 5 2023.0000 10.2000 Evidence-based medicine 14 6 2023.0000 5.1667 Students 14 6 2023.0000 8.5000 Artificial intelligence and writing 14 3 2023.0000 56.0000 Medical 10 3 2023.0000 2.6667 Patient communication 10 3 2023.0000 2.0000 Stroke 10 3 2023.0000 0.6667 Medical informatics 9 4 2023.0000 2.5000 Cluster 6: The Utilizations and Issues of ChatGPT in Medical Education and Research Artificial intelligence 1578 488 2023.0287 5.4242 Education 100 26 2023.0385 12.4231 Research 93 24 2023.0000 5.5417 Health 39 11 2023.0000 2.0000 Public health 38 10 2023.0000 6.2000 Bias 30 8 2023.0000 5.8750 Higher education 26 7 2023.0000 3.8571 Peer review 26 6 2023.0000 1.1667 Scientific writing 24 7 2023.0000 22.4286 Covid-19 23 8 2023.0000 3.1250 Academic integrity 21 5 2023.0000 5.8000 Precision 18 3 2023.0000 1.3333 Scientific research 17 5 2023.0000 1.6000 Misinformation 16 4 2023.0000 7.5000 Methodology 15 3 2023.0000 9.3333 Teaching 14 3 2023.0000 0.0000 In conclusion, we believe that our study provides new quantitative evidence and methodological insights into the global landscape and hotspot evolution of generative AI-related research in the medical field. We thank Tang and colleagues for their valuable comments and believe that further research is still necessary for continuous improvement in terms of wider coverage of data sources, sensitivity analysis of threshold settings, evaluation of the validity of clustering structure, and keyword normalization. At the same time, we are confident that our study, as one of the earlier attempts to combine machine learning topic identification with external validation across databases, provides a feasible path to improve the robustness and reproducibility of the bibliometric analysis and is expected to provide a reference for subsequent studies in terms of research trend monitoring and evidence integration. We are looking forward to continuous communication and cooperation with colleagues in the academic community to promote the deepening application and high-quality translation of bibliometric methods in emerging cross-cutting fields.