Current Practices in Quality Assessment of Systematic Reviews in Computing: Exploring Automation Potential

Systematic Literature Reviews (SLRs) play a crucial role in evidence synthesis within computing research. However, the quality of SLRs can vary significantly, affecting their reproducibility and trustworthiness. This study addresses the problem of poorly understood practices in SLR quality assessment. It investigates the current landscape of quality instruments used to assess SLRs in computing by analyzing 97 tertiary studies across various computing domains. The analysis focuses on identifying the dominant quality instruments, and examining reported modifications or adaptations made to them. A qualitative analysis is conducted on the interpretations and scoring of widely utilized quality criteria. The analysis reveals diverse interpretations and potential inconsistencies in the application of quality instruments, owing to the absence of concrete examples. The findings provide valuable insights for both SLR authors and consumers in computing research, pointing out the most widely deployed quality instruments, common customization and interpretive practices, and potential areas for improvement. This study contributes to the ongoing discussions on enhancing SLR quality in computing, forming the basis for automating the quality assessment process.