The effect of manuscript characteristics on quality assessment of scientific manuscripts in psychology

We report a vignette study and a survey to investigate which study characteristics influence quality ratings psychology academics give of articles submitted for publication, and psychology academics and students to theses written by psychology students. In the vignette study, 800 respondents evaluated the quality of an abstract (7-point scale) of studies with small ( N = 54) or large sample sizes ( N = 540), showing statistically significant or non-significant results, and containing statistical reporting errors or no errors. In the subsequent survey, the same participants rated the importance of 29 manuscript characteristics related to a study’s theory, design, conduct, data analyses, and presentation for assessing either the quality of a manuscript or its publishability (article) or grade (thesis). Results showed that quality ratings were affected by sample sizes but not by statistical significance or the presence of statistical reporting errors in the rated research vignette. Furthermore, in the survey academics and students provided highly similar ratings of the importance of different aspects relevant to quality assessment of articles and theses. These results suggest that quality criteria for scientific manuscripts are already adopted by students and are similar for submitted manuscripts and theses. Further research is needed to examine which study characteristics researchers consider most important when assessing a manuscript’s quality in peer review.