On the Relevance of Paper-Type Information in Systematic Mapping Studies in Software Engineering

Systematic Mapping Studies (SMSs) are valuable in evidence-based software engineering research. SMSs aim to provide an overview of research, identify gaps and trends, and assess the feasibility of conducting a more focused systematic literature review. In current guidelines for conducting SMSs, a quality assessment of the included papers is suggested only when the research questions explicitly require such a quality assessment. We agree with the recommendation that quality assessment is generally non-mandatory. However, SMSs deal with papers ranging from opinion papers to papers reporting highly rigorous empirical studies. Therefore, in this paper, we argue that analyzing the type of papers is essential for almost every intended purpose of an SMS. Otherwise, without distinguishing papers based on their types, we risk deriving a less informative or incomplete overview or, at worst, a misleading overview of research. Petersen et al. ‘encourage’ the classification of papers into six paper types as proposed by Wieringa et al.: evaluation research, solution proposal, validation research, philosophical papers, opinion papers, and personal experience papers. Given the lenient guidelines on assessing the quality of included studies, we recommend a stronger focus on classifying papers by type.