The role of performance level descriptions for the effects of rubrics on self-assessment accuracy and performance

The experiment followed a 2 × 2 factorial between-subjects design where the factors rubric vs. rating scale during self-assessment and rubric vs. rating scale during revision were varied, which resulted in four conditions. All learners read a scientific article and were instructed to write an abstract for this article. At the end of the writing phase, learners were asked to self-assess the quality of their abstracts on six assessment dimensions. During self-assessment, based on random assignment, the learners received either the rubrics or the rating scales. Then, learners decided on which of the six assessment dimensions they would like to receive further instruction (i.e., learners took regulation decisions). They could choose as many dimensions as they wanted to. Afterwards, they were to revise their abstracts on the chosen dimensions. During revision, based on random assignment, we again varied whether learners were supported by rubrics or only received the rating scales. We determined the required sample size for the experiment by running an a priori power analysis with G*Power (Faul et al., 2007 ). Based on the minimum desired effect of instructional interventions of d = 0.40 according to Hattie ( 2009 ), we used a targeted effect size of d = 0.40/ f = 0.20 in this analysis. The further parameters were α = 0.05, and β = 0.20 (i.e., power of 80%). In a 2 × 2 factorial between-subjects design, the power analysis indicated that a sample of 199 participants would be required to detect the effects at which our research questions were targeted. We slightly oversampled and recruited a total of N = 236 German university students (186 female; M Age = 22.71 years, SD Age = 2.77 years). University students were bachelor’s and master’s students from different subjects. Bachelor students accounted for 48.3% of the sample, master students for around 51.7%. The majority of participants was enrolled in Humanities or Social Science programs which also included preservice teachers (87.3%), followed by students from the domain of Law or Architecture (5.1%), Natural Sciences (3.8%), Technology and Engineering (3.0%), and Economics (0.8%). The voluntary participation was compensated with €20 and written informed consent was collected from all participants. This study’s design and the analyses were not pre-registered. The study was approved by the ethics committee of Ruhr University Bochum (EPE-2022-35). All materials were provided in a computer-based learning environment. The materials were based on the material used by Krebs et al. ( 2022 ). All materials were in German and an expert in the field of German didactics (i.e., the third author of this study) supervised the entire material development. All learners received an introduction in how to write a scientific abstract. This introduction was provided in written form with short explanations and bullet points. First, learners were informed about what an abstract is. Based on Swales and Swales ( 1990 ), Swales and Feak ( 2009 ), and Busch-Lauer ( 2012 ), the learners were informed that abstracts are short summaries of a scientific article and should give the reader an overview about the empirical study that is presented in the article. Moreover, they were instructed that an abstract should contain the most important aspects of the study (for empirical research: research questions or hypotheses, the data basis, method, results, and discussion). Afterwards, they received information about how to write a scientific abstract. This information was structured around six assessment dimensions, which were the same dimensions on which learners were to self-assess their performance later: (a) content, (b) formal, (c) orthography and grammar, (d) scientific style, (e) text organization, and (f) structure. For each dimension, learners were informed about the important assessment criteria that needed to be met (these criteria were also part of the rubrics and rating scales, see section Rubrics and Rating Scales During Self-Assessment and Revision ). For instance, for the dimension of (a) content, the learners were informed that research questions/hypotheses, the data basis, the method, the results, and a brief conclusion of the results needed to be included. To help learners discern and understand the six dimensions, they received contrasted exemplars (one of high quality and one of low quality) for each dimension. These exemplars illustrated the assessment criteria of the dimensions, but their content was not related to the content of the scientific article. All participants read a slightly adapted scientific article that was published under the title “Steigert Kaugummikauen das kognitive Leistungsvermögen? Zwei Experimente der besonderen Art ” [ Does Chewing Gum Enhance Cognitive Abilities? Two Rather Extraordinary Experiments ] by Rost et al. ( 2010 ). The article consisted of 1,810 words and was fitting the criteria of a well-written scientific article. It was slightly adapted so that it was understandable without prior knowledge on statistics. The learners were asked to read the article and afterwards were instructed to write an abstract for it. Learners were prompted to keep in mind what they had learned in the introduction about the criteria that an abstract should meet. During writing, which was limited to 20 min, the article was available and a timer showing the remaining time was displayed. After the writing phase, all learners were required to self-assess the quality of their abstracts. Learners in the conditions who received the support of rubrics during self-assessment were given a specific rubric for every dimension (i.e., six in total) separately, and these were displayed to them one after another. Each rubric contained the two main components a rubric is made of, that is, assessment criteria and performance level descriptions as well as a rating scheme (from 1: very poor quality, to 6: very good quality). Hence, the learners were to provide one self-assessment for each of the six assessment dimensions (resulting in six self-assessment scores). The learners with only the rating scales during self-assessment also self-assessed their performance on the six dimensions. However, in contrast to the conditions that were supported by rubrics, they were presented only one of the main components of rubrics, that is, they only received the assessment criteria and a rating scheme (which was the same as for learners with rubrics, ranging from 1 as very poor quality to 6 as very good quality). Similar to the learners with rubrics, they were asked to assess themselves on each dimension separately, resulting in six self-assessment scores as well. Figure 1 shows the self-assessment phase for learners with rubrics vs. for learners with rating scales for the dimension of text organization. Screenshots from the self-assessment phase in the learning environment for all dimensions can be found in Appendix A. During self-assessment, in all conditions learners could see their self-written abstracts but could not modify them. Self-assessment with rubric vs. with rating scale for the dimension of text organization (translated from German) Following the self-assessments, learners were to take regulation decisions. Specifically, all learners were instructed: “Now it’s time to improve your abstract. You now have the option of displaying further exemplars of the dimensions you have already learned about and then revising your abstract. You can choose between zero and all six dimensions. Please note: the number of dimensions you select does not affect the duration of the study; the study will always be the same length. Please now click on all dimensions for which you would like to receive further information and which you would like to improve.” Based on their regulation decision(s), the learners received two exemplars for each chosen dimension (one of high and one of low quality) and were instructed: “Try to understand why the exemplar fully meets/does not fulfill the assessment criteria of the dimension content/formal/orthography and grammar/scientific style/text organization/structure !” The exemplars for the dimension of text organization are shown in Fig. 2 . Exemplars for the dimension of text organization (high and low quality) that were shown to the participants after the regulation decision (translated from German) Immediately after studying the two exemplars for a dimension, the learners were asked to revise their abstract concerning the respective dimension with the following prompt: “Please revise your abstract here with regard to the dimension content/formal/orthography and grammar/scientific style/text organization/structure!” During this revision phase, we varied whether learners were supported with rubrics or only received the rating scales (the rubrics and rating scales were identical to the ones used during self-assessment). There was no second self-assessment phase at the end of the revision phase. We measured task performance at two time points, that is, after the initial writing phase and after the revision phase. For this purpose, two trained raters assessed the quality on each of the six assessment dimensions using the same rubric the learners received. Between 1 (very poor) and 6 (very good) points could be awarded for each dimension. Interrater reliability, determined by the intraclass correlation coefficient with measures of absolute agreement, was very good (for all dimensions, ICC > 0.85). The scores of all six dimensions were aggregated, resulting in a minimum of 6 points and a maximum of 36 points as an overall performance score. To compute the accuracy of learners’ self-assessments, we compared the six assessments of the learners with the assessments of the experts after the initial writing phase and calculated three different measures. First, we calculated the absolute discrepancy between learners’ and experts’ ratings (|X learners’ assessment - X experts’ assessment |), that is, absolute accuracy (see Schraw, 2009 ). This measure describes the degree of correspondence between two ratings and allows values between 0 (i.e., perfect accuracy) and 5 (i.e., max. inaccuracy) for each dimension. We aggregated these scores by summing them up over the six dimensions (i.e., scores between 0 and 30). Second, we calculated bias , which is the signed difference between the learners’ assessments and the experts’ ratings, allowing to account for over- or underconfidence. Bias was calculated by first subtracting the experts’ assessment from the learners’ assessment (X learners’ assessment - X experts’ assessment ) separately for each dimension (six scores between − 5 and 5) and averaging the six scores afterwards. As a result, we receive one value for each learner that varied between − 5 and 5. Here, positive values indicate that the learner overestimated their performance, values close to zero indicate high accuracy, and negative values signal underconfidence. In order to assess the processes in which the learners engaged during self-assessment and hence to be able to test our hypothesis that learners with rubrics would establish more meaningful relations (i.e., relations of better quality) between the degree to which the assessment criteria are met and different performance levels (Hypothesis 1b), learners were instructed to think aloud. Specifically, after a brief introduction to the think-aloud methodology, based on recommendations by Ericsson and Simon ( 1993 ) the learners were prompted: “[…] Please say out loud everything that comes to your mind and goes through your head while you are solving the task. It is important that you do not try to explain or structure what you are doing. Just imagine that you are alone in the room and talking to yourself. […]”. These think-aloud protocols were analyzed using the following coding procedure. First, as a proxy for whether learners attended to predictive cues, we determined the number of the assessment criteria (these were included in both the rubrics and the rating scales) that learners mentioned. Second, we determined how learners related the degree to which the assessment criteria of a dimension were met to the respective performance levels by rating their quality using a 3-point rating scale. As learners in both conditions were provided with a rating scheme that included six performance levels, these relations could be established by all learners. However, only the learners who received rubrics during self-assessment also received performance level descriptions and hence establishing meaningful relations between the assessment criteria and performance levels was deemed more likely for these learners to occur. Specifically, the quality of these relations to performance levels was rated as high (i.e., a score of 2) if the self-assessment processes were justified in a differentiated manner, that is, performance level descriptions or scale points were mentioned and set in relation to the self-written abstract and/or further differentiated explanations were given to why this assessment was made. The quality of relations to performance levels was rated as medium (i.e., a score of 1) if the self-assessment processes were rudimentarily justified, that is, the performance level descriptions/scale points were mentioned but not further elaborated, and no further explanations were given as to why this assessment was made. A low-quality rating of relations to performance levels (i.e., a score of 0) was provided when self-assessment processes were not justified, that is, performance level descriptions or scale points were not mentioned or differentiated, the criteria were not weighted against each other, and no further explanation was given as to why the specific assessment was made. The quality scores were determined for the six dimensions separately and aggregated into an overall score. Interrater reliability was very good for counting the utilization of predictive cues (ICC > 0.94) and for rating the quality of the relations to performance levels (ICC > 0.83). We measured learners’ subjective cognitive load at five different time points during the experiment: after reading the scientific article, after writing their abstract, after self-assessment, after the regulation decisions and after the revision phase. Specifically, on each measurement time point, we asked the learners to rate their perceived difficulty and mental effort on 7-point Likert scales ( 1: very low difficulty/mental effort , 7: very high difficulty/mental effort ). The wording of the questions was adapted from previously existing scales for the assessment of mental effort and perceived difficulty in task processing (Kalyuga et al., 1999 ; Paas, 1992 ; Paas et al., 2003 ). Instructional efficiency was measured according to the measure introduced by Paas and Van Merriënboer ( 1993 ), which combines measures of test or task performance with measures of subjective cognitive load required to attain this performance (see also Van Gog & Paas, 2008 ). Specifically, we used the improvements of task performance (i.e., performance gains) the learners realized during the revision phase, standardized them and combined them with learners’ ratings of perceived difficulty and mental effort (these measures were averaged) during the revision phase according to the formula suggested by Paas and Van Merriënboer ( 1993 ), \(\:\frac{{zP}_{Improvement}-{zE}_{Cognitive\:Load}}{\sqrt{2}}\) . All learners worked individually in a digital learning environment in a laboratory in soundproofed cubicles to avoid distractions. All learners were asked to give general demographic information on gender, age, GPA score and university semester. Then, all learners received the thematic introduction in how to write a scientific abstract, which included the different dimensions and assessment criteria. Afterwards, they read the scientific article. Reading time was held constant at 20 min. Next, they wrote their abstract for the article. Writing time was held constant at 20 min. In the next step, the took Learners were to one of the four conditions and were asked to self-assess their abstracts concerning the six dimensions, either with the help of rubrics or rating scales. We prompted all learners to think while their Following the self-assessments, the learners took their regulation decisions and were prompted to think while they had their regulation learners the exemplars of the chosen dimensions and were to revise their abstracts was separately for each dimension, learners could in that dimensions as either with the help of rubrics or rating scales, in a In this phase, rubrics and ratings scales were only but no assessment was required from the learners. After their for the dimensions one by the experiment The experiment took A overview of the can be found in Fig. . of the of was used for all analyses the analyses concerning see 1 and 2 the scores and for all on all measures of the and analyses can be found . All analyses were with To test for a priori between the we and To our the effects on self-assessment accuracy we used as well as a measures to account for between the dimensions (Hypothesis to the which the performance level descriptions affect self-assessment accuracy, we a analysis with from using the 2022 ; see To test our concerning the effects on performance in the revision phase we again used and to account for the of results to data was not which is the for the slightly different of in the As the data related to only five participants (i.e., of the we not try to the data points. We not from our In the first step, we whether the random resulted in that were in of learning not concerning age, = = or = = the not with to gender, = = the not in of study = = or in the = = what an all In the second step, we whether were between the concerning the reading phase and the initial phase of writing the abstract, which both took the were In the reading phase, were no concerning both measures of subjective cognitive = = = for perceived and = = = for mental were no concerning cognitive load in the initial phase of writing the abstract, = = = for perceived and = = = for mental the not concerning task performance in the initial writing phase (i.e., the overall quality of their abstracts at the end of the initial writing = = = these results indicate that the random resulted in 1: is the effect of performance level descriptions on self-assessment 1: is the effect of performance level descriptions on self-assessment In we that learners with rubrics (i.e., assessment rating and performance level would self-assessment accuracy in to them with only rating scales (i.e., assessment criteria and a rating We found a difference between the learners with rubrics and the learners with rating scales in of absolute accuracy, = = d = As the learners with rubrics better absolute accuracy (i.e., values to their ( M rubrics = SD = M rating scales = SD = In of we not a difference between the learners with rubrics and with rating scales, = = d = a basis, learners with rubrics a ( M rubrics = SD = learners with rating scales ( M rating scales = SD = In we that the of the rubrics over the rating scales concerning self-assessment accuracy would be through the rubrics learners to establish more meaningful relations (i.e., through their quality between the degree to which the assessment criteria were met and the performance The results of the analysis the effect on absolute accuracy are shown in Fig. . We found a effect the quality of the established relations between the degree of how the criteria were met and different performance levels, = This that to the of the quality of the established the learners with rubrics had an of in of absolute accuracy, that with an of one in quality, learners with rubrics had a more learners with only the rating scales. that the effect in was only for absolute accuracy, we from using bias as the of the analysis concerning the effect on absolute accuracy For we also analyzed whether learners with rubrics with rating scales during self-assessment would affect cognitive load in However, were no between the in of perceived difficulty during self-assessment, = = d = or mental effort during self-assessment, = = d = are the effects of performance level descriptions on task performance and instructional efficiency in a revision are the effects of performance level descriptions on task performance and instructional efficiency in a revision we our performance in the revision phase, we whether learners had their performance in the revision phase. we found that learners realized performance = = the performance on the dimensions the learners for revision from = d = Next, we our In we that learners with rubrics they in revision would to performance in a revision phase (Hypothesis However, a 2 × 2 factorial between-subjects with the factors rubrics vs. rating scales during self-assessment and rubrics vs. rating scales during revision not a main effect of the of rubrics vs. rating scales during self-assessment , = = the of rubrics vs. rating scales during self-assessment not affect the level of cognitive load during regulation in of perceived = = in of mental = = In we that learners with rubrics during revision would not performance in a revision phase in to learners with rating scales. There was no main effect of the rubrics vs. rating scales during revision performance in the revision phase, = = was no between the two To more test whether rubrics and rating scales resulted in performance in the revision phase, in the next we an Based on recommendations by ( ), we an effect size of d = as the and an effect size of d = as the for effect of To test for we determined the of the effect size of the difference between the with rubric and with rating scales concerning performance which was − to As the included a value our positive it can be that the performance were not with the of the is hence no difference but also no between rubrics and rating scales concerning performance in the revision phase. Similar to our performance no effects were when instructional the performance in relation to the cognitive load in the revision phase, was targeted (Hypothesis but this time to our a 2 × 2 not indicate a main effect of the of rubrics vs. rating scales during revision , = 0.20 = This of results not for only cognitive load in the revision phase and hence not the performance in relation to cognitive load (i.e., instructional were There were no between the in of perceived difficulty during revision, = = d and mental effort during revision, = = d = We also analyzed whether the effects of the two factors on performance in the revision phase would on learners’ performance in the initial writing phase. This analysis that the effect of the rubrics vs. rating scales during self-assessment on performance in the initial writing phase, = = = of this effect with the using the that for learners with initial task performance of which to five participants (i.e., of the rubrics during self-assessment a effect on performance in the revision phase in to rating scales. for learners performance in the initial writing phase which to learners (i.e., of the rubrics during self-assessment performance in the revision phase. For we also analyzed whether the would be when only the performance on the dimensions the learners to revise were for the on overall performance in the revision phase, were no main effects or between the all However, again the effect of rubrics vs. rating scales during self-assessment on learner performance in the initial writing phase, = = of this effect that for learners with initial task performance of which to participants (i.e., of the rubrics during self-assessment a effect on performance on the chosen dimensions in the revision phase. for learners performance in the initial writing phase which to learners (i.e., of the rubrics during self-assessment performance in the revision phase.