Construction, editorial practice, and AI-assisted evaluation of the Chinese Medical Research Reporting Specification Checklist System

With the development of academic publishing and the iteration of artificial intelligence (AI) technology, the submission volume of Chinese journals has been increasing continuously, which imposes an increasingly heavy workload for peer review, editorial processing, and proofreading in academic journals. This study aimed to construct a localized, easy-to-operate, and efficient Chinese Medical Research Reporting Specification Checklist System (referred to as the Disclosure Checklist), apply it to editorial practice, and explore the accuracy, consistency and stability of AI Large Language Models (LLM) in the review of disclosure items. Based on the 10 core reporting checklists of the guidelines from the EQUATOR Network, we optimized them using the Delphi expert consultation method to better adapt to the Chinese context, and classified their recommendation levels using a 5-point Likert scale, and finally optimized into 9 types. The Kimi 2.5 fast model was used to review the reporting items in the original manuscripts and published manuscripts of two journals, and the paired t -test and Wilcoxon signed-rank test were used to compare the improvement in reporting completeness between the two versions of manuscripts. Different research types of papers were selected and independently reviewed three times by the Kimi 2.5 fast model. The Friedman test was used to compare overall accuracy across the three review rounds, with post-hoc pairwise comparisons performed using the Wilcoxon signed-rank test at the level of study-design categories, and the Krippendorff's Alpha, and Gwet's AC1 tests were used to assess the consistency and stability of the three sets of results. R version 4.5.2 was used for statistical analysis. A total of 9 types of Chinese medical research reporting guideline checklists were optimized, including 281 items. The evaluation of 115 articles published after revision using the 8 types of checklists (meta analysis checklist was excluded) revealed that reporting quality scores were 4.12 points (95% CI: 3.42–4.82) higher in the published version than in the original manuscript, with the lower confidence limit indicating a minimum improvement of 3.42 points, a difference that is clinically meaningful for editorial quality assessment [Cohen's d = 1.08, Rosenthal's r = 0.736(0.639, 0.810)]. The Kimi 2.5 fast model reviewed 900 checklist disclosure items across 8 types of biomedical papers, with manual verification. The overall accuracy of the three rounds of review ranged from 72.67% to 75.78%, and did not differ significantly across rounds, while Krippendorff's Alpha was 0.635 (95% CI: 0.602–0.669), and Gwet's AC1 was 0.663 (95% CI: 0.631–0.691), all falling within the "moderate-to-substantial" agreement range. The Kimi 2.5 fast model shows certain potential in assisting the review of disclosure items in Chinese medical research articles, but optimization for error-prone items to further improve accuracy and consistency is still required. This study provides support for the future development of domain-specific intelligent agents for disclosure checklist review in Chinese medical journals. It is expected that in the near future, stable, accurate and reliable review AI models can be embedded into journal submission and peer review systems, so as to improve editorial efficiency, reduce the workload of reviewers, and enhance the transparency and quality of papers.