Validity of Simulation Studies: A Case Research in the Context of Differential Item Functioning Detection

The aim of this study is to examine the simulation validity by determining whether the simulation process produces results that are realistically close to expectations, through the generation of artificial data containing Differential Item Functioning (DIF) and assessing whether the data were accurately generated. In the study, which involves one reference group and two focal groups, 2250 different conditions were simulated by considering factors such as the sample size of the reference group, the sample size ratios of the focal groups, the amount of DIF, and the DIF technique. During the data generation process, random data for difficulty and discrimination parameters were generated using the Two-Parameter Logistic Model (2PLM), and it was planned that 20% of the items in the test would contain DIF. To test the validity of the simulation, mean absolute bias and RMSE values for the difficulty and discrimination parameters were calculated both at the item level and by considering the relevant factors. The analysis results revealed that the mean absolute bias and RMSE values calculated for the difficulty and discrimination parameters were low and close to zero. This indicates that estimation errors were minimal and supports the validity of the results. Additionally, it was found that the sample size of the reference group and the sample size ratios of the focal groups had a statistically significant effect on the mean absolute bias and RMSE values for both difficulty and discrimination parameters, and it was observed that as the sample size increased, the mean absolute bias and RMSE values decreased. However, it was concluded that the amount of DIF added to the focal groups did not have a significant effect on the accuracy of parameter estimations. The findings demonstrate that sample size plays a critical role in the accuracy of parameter estimations, while the amount of DIF does not significantly impact this process, and the results of the study are consistent with relevant research in the literature. As a result of this research, it has been recommended that validity evidence for the simulation should be provided not only in DIF investigation studies but also in simulation studies conducted in various subject areas within the field of psychometrics.