Assessing the Assessments: Quality Metrics for Short Answer Questions and Essay Type Questions

INTRODUCTION Measuring and guiding learning is important during medical assessment. With competency-based education in India, multiple-choice questions (MCQs) are becoming more prominent because of their objectivity, simple scoring mechanism, and psychometric evaluation.[1-3] Being a practice for questions, students will always find it difficult with these typically labeled Short Answer Questions (SAQs) and Essay Questions. Hence, these types of questions would target factual know – how plus the ability to analyze, think critically, argue professionally, and participate in ethical judgments.[4,5] Nevertheless, the main criticisms of SAQs and EQs stem from their perceived subjectivity, variability in scoring, and difficulty in achieving reliability across examiners and student cohorts.[6,7] Item analysis is a procedure carried out in MCQ-based assessments that evaluates each individual question based on data of student performance. It comprises measures such as difficulty index, discrimination index, and internal consistency.[8] This can be extended to open-ended questions to give more psychometric soundness to SAQs and EQs.[9,10] Medical educators are currently analyzing test items for the sake of learning and meaningful feedback.[11,12] This study aims to analyze item analysis methods on SAQs and EQs, assessing psychometric qualities such as difficulty, discrimination, reliability, and internal consistency, to create valid assessment tools in line with competency-based medical education (CBME) and National Medical Commission’s regulations. MATERIALS AND METHODS The sample consisted of 120 students of the 2nd MBBS professional term of the CBME system. In view of the ethical considerations, anonymity was ensured. The paper consisted of short answers and essay-type questions. Each question carried 10 marks for a total of 80 marks. The marking scheme involved a structured answer scheme for analytic rubric on content accuracy, and markings were done by a pair of subject experts. The classroom assessments were analyzed for items with difficulty index, discrimination index, inter-rater reliability, and internal consistencies. The discrimination index was supposed to be acceptable if it was 0.25 or higher, and Cohen’s Kappa was used for inter-rater reliability. All psychometric values were compared to established benchmarks in educational measurement literature to assess the quality of the items.[1-5] Ethical clearance Study approval was granted by the Institutional Ethics Committee of N. C. Medical College and Hospital, Israna, Panipat, Haryana, India, under letter no. NCMCH/EC/2022/27 dated March 10, 2022. RESULTS The scoring rubric that was used to evaluate the SAQs in the study. The rubric contained four levels of satisfactory and unsatisfactory performance. The scoring was from 0 to 3. The highest score, 3, belonged to those answers that were complete, accurate, and well articulated; 2 went to those with at least the principal points, although slight omissions were present: Very small factual errors might undersell some. A score of 1 was given to answers that either showed limited understanding or were too brief to really demonstrate any significant insight. However, answers that were incorrect or left blank were rated with a score of 0. Thus, the rubric allowed for a standardized and objective evaluation of student performance. The analytic rubric used for grading essay questions (EQs). Thus, the rubric assessed four domains, with the highest cumulative score given as 10. Content accuracy was the most significant criterion, carrying 4 points, whereby factual correctness and alignment with learning objectives were evaluated. Organization and structure, allocated 2 points, were assessed for the logical progression and coherence of arguments, as well as apt paragraphing. Depth of explanation, carrying 2 points, reflected a student’s ability to apply knowledge to reasoning that is beyond surface level, providing insights from critical reasoning. Clarity and language (2 points) evaluated grammar, appropriate use of terminology, and overall clarity of expression. The structured rubric enabled the assignment of objective criteria to essay responses throughout the fair and comprehensive framework of assessments. An overview of the assessment and item analysis process adopted in the study. After conducting the tests among students, which consisted of 10 SAQs and 5 essay questions (EQs), the marking was done independently by two examiners for each response. Their scores were moderated whenever they differed by more than 10%. The agreed-upon moderated marks were then entered into an Excel worksheet. The item analysis was then carried out in SPSS version 22, which computed the difficulty index, discrimination index, inter-rater reliability (Cohen’s Kappa), and internal consistency (Cronbach’s alpha). Such a method allowed for fair scoring and rigorous investigation into the worthiness of the tests. The analysis of 10 SAQs and 5 EQs in the written assessment gave insight into the psychometric characteristics concerning difficulty level, discrimination power, inter-rater agreement, and internal consistency. The difficulty index and the comparison with EQs were studied for constructed-response assessments (SAQs). SAQs had a difficulty index mean of 62.5%, whereas EQs had a range of 45%–70% of difficulty. The mean discrimination index for SAQs was 0.31, thus denoting strong differentiation between the better-performing and the weaker-performing students. Inter-rater reliability level obtained is moderate, with Cohen’s Kappa at 0.71 for SAQs and 0.62 for EQs. Internal consistency measure, Cronbach’s Alpha, held an acceptable α-for SAQs of 0.78 at the level of the classroom. The study showed that SAQs outranked EQs for discrimination indices, inter-rater reliability, and internal consistencies. This stronger correlation of scores could be linked to the structured nature of SAQs and the existence of a pre-established rubric for scoring. On the contrary, EQs score on subjectivity where their items are phrased variably. The results indicate that item analysis may improve the quality of constructed-response assessments. DISCUSSION The analyses of SAQs and EQs revealed psychometric characteristics and their assessment properties. Perhaps due to the rigid structures, SAQs have higher discrimination and reliability indices. An adequately designed analytic rubric might make scoring more objective and consistent.[4,7] The study contends that essay questions (EQs) have less inter-rater reliability due to elongated verbal responses and interpretations. Variation in EQ performance occurs generally due to factors concerning the content and construction of prompts. If well structured, clinically relevant, and accompanied by specifications of learning outcomes, an EQ should perform well. The study highlights the urgent need for well-developed rubrics and faculty training for the implementation of EQs.[5,9] These analyses fall in line with current literature, further asserting the fact that an analytic rubric and trained raters are required to reduce inherent subjectivities in constructed-response question evaluation.[6,11] Testing is essential for reliability, transparency, and fair evaluation of students. Most SAQs and EQs can discriminate between higher- and lower-level students, but some of them do not have this power of discrimination. A lack of discriminatory power leads to inconsistent measurement of performance. Therefore, item review and prevalidation must be done so that those that are sound can be kept, and those that are poor, revised, or eliminated from the tests. Such a practice finds wide endorsement throughout test development literature, especially in multiple-choice and item banking methodology.[8,10] To summarize the literature, this research establishes that psychometric evaluation is essential to routine assessment. It thereby supports continual faculty development pertaining to rubric design and assessor training. The periodic improvement of assessment instruments and scoring methods affords institutions the environment wherein the validity, reliability, and educational purpose of the assessment may be maintained. CONCLUSION Constructed response exam items (SAQs and EQs) are very important in medical education assessment, whereby they test the analytical reasoning, integration of knowledge, and communicating of clinical information by students. However, these exercises are subject to criticism for their alleged subjectivity and idiosyncrasy in scoring. A study points out that these items have inter-rater reliability and discrimination comparable to those of objectively scored formats when supported by structured analytic rubrics and subjected to routine item analysis. The study recommends a psychometric validation approach to upgrade the quality of fair assessments to contribute to CBME. Limitations Scope of Single-Center Study: The study was conducted within a single academic institution setting. Hence, one cannot expect all conclusions to be transferable to other contexts or institutions wherein assessment practices differ Small Number of Extended Questions: A relatively small number of EQs were taken into study, thus weakening conclusions about their psychometric properties Methodological Limitations: It is based on a classical test theory approach, which though popular does have inherent limitations. Perhaps future studies will be benefited by looking from the perspective of modern psychometric techniques such as item response theory or generalizability theory. Recommendations The governing bodies should establish regular analyses of SAQ and extended questions to be done by the assessment quality assurance agencies. This helps to identify poorly performing items, create evidence for revision, and sustain a decent-worthy bank of items for student assessments. Further training of faculty in rubric-based methods to assess SAQs and EQs is also recommended. This will improve transparency in grading practices, complement professional development, and support academic integrity. The training may be linked with other development activities or even be required for exam preparation. Financial support and sponsorship Nil. Conflicts of interest There are no conflicts of interest.