A Comparative Study of Difference-in-Differences (DID), Synthetic Control Method (SCM), Synthetic Difference-in-Differences (SDID): Effects of Early-Morning Delivery

Purpose: The purpose of this study is to compare and analyze the strengths and weaknesses of three widely used quasi-experimental methods in causal inference research: Difference-in-Differences (DID), Synthetic Control Method (SCM), and Synthetic Difference-in-Differences (SDID). The study specifically examines the impact of Early-Morning delivery services on existing offline commercial districts as a case study to empirically evaluate the accuracy and applicability of each method. Research design, data, and methodology: To compare the three methodologies DID, SCM, and SDID this study analyzed the impact of Early-Morning delivery services on offline supermarket sales. The treatment group consisted of the Seo-gu and Yuseong-gu districts in Daejeon, while the control group included selected districts in Gwangju and Busan. Using daily credit card panel data, the analysis covered sales changes from May 2, 2019, two years before the entry of dawn delivery services, to October 31, 2021, six months after their introduction. The DID method compared post-treatment sales changes between the treatment and control groups. The SCM method created a synthetic control group by assigning weights to the control group to closely match the treatment group during the pre-treatment period, then compared post-treatment sales changes between the treatment and synthetic control groups. Similarly, the SDID method generated a synthetic control group with the weights of SCM and the time weights to minimize differences between the pre- and post-treatment periods, and then compared post-treatment sales changes between the treatment and synthetic control groups. This approach allowed for the analysis of treatment effects based on differences observed after the intervention among the treatment, control, and synthetic control groups. For analyses based on the number of control groups, selections were made in both similar and dissimilar orders by comparing the pre-treatment trends of Seo-gu and Yuseong-gu in Daejeon with those of potential control groups using Euclidean distance. The comparative analysis based on the pre-entry period fixed the post-treatment period to 6 months after the entry of the dawn delivery service, while the pre-treatment period was divided into 2 years, 1 year, 6 months, and 3 months before the entry. This setup allowed the study to examine cases where the pre-treatment period was longer than, equal to, or shorter than the post-treatment period. Results: The analysis results indicated that the introduction of Early-Morning delivery services had a negative impact on offline supermarket sales across all three models: DID, SCM, and SDID. However, the SCM and SDID models, by constructing synthetic control groups, maximized the similarity between the treatment and control groups, resulting in more reliable counterfactual estimates and providing more stable estimates than the DID model. This is because DID may produce biased results if the pre-trends of the treatment and control groups are not similar. Specifically, the SCM model delivered stable results even when the number of control groups was small and when the parallel trend assumption was not met. The SDID model produced more stable results as the pre-intervention period extended. Conclusions: In conclusion, the DID model is appropriate when the parallel trend assumption is satisfied; however, when this assumption is not met, the SCM or SDID models are more suitable. SCM and SDID are particularly effective in scenarios where the parallel trend is not satisfied or the number of control groups is small. Additionally, the SDID model has proven to be useful for analyzing long-term data. The study highlights that the performance of each method can vary depending on the situation, emphasizing the importance of selecting the appropriate methodology based on the specific circumstances of the analysis. Consequently, these models contribute to more accurate causal inference in policy evaluation and can aid in decision-making and strategy development across various fields.