An R Package for Optimizing the Composite Reliability in Multivariate Nested Designs

Background The reliability of assessment tools is a crucial aspect of monitoring student performance in various educational settings. It ensures that the assessment outcomes accurately reflect a student's true level of performance. However, when assessments are combined, each assessment contributes as a data point towards the overall performance of the student on the overarching educational framework. In that case, determining composite reliability can be challenging, especially for naturalistic and unbalanced datasets, as is common in programmatic assessment. Results This paper introduces the CompositeReliability package in R, designed to estimate composite reliability using multivariate generalizability theory and enhance the analysis of assessment data. The package produces extensive Generalizability and Decision study results with graphical interpretations. Composite reliability incorporates weights and covariance to integrate results across assessment tools. Weights can be optimized to minimize standard error of measurement or maximize reliability. Conclusions Overall, the package’s flexible use and optimization empower assessment tailoring and robust insights into student performance. The approach is suitable for programmatic assessment. The package facilitates reliable, comprehensive evaluation across diverse assessments.