Penalized GEE for Complex Carry‐Over in Repeated‐Measures Crossover Designs

Crossover designs are commonly employed in clinical and behavioral research, yet the statistical models used to analyze them often rely on unrealistic assumptions-either ignoring carry-over effects or modeling them as simple and homogeneous across treatment sequences. However, carry-over effects are frequently complex, varying by treatment order and interaction, and until now, no statistical methodology had been formally established to estimate such complex effects. This paper introduces a penalized semiparametric Generalized Estimating Equations (GEE) approach designed to estimate first order complex carry-over effects in crossover designs with repeated measurements. We first derive identifiability conditions under which complex carry-over effects become estimable. We then provide theoretical guarantees-building on an extension of the sandwich variance formula-showing that the proposed penalized estimator achieves asymptotic normality for the functional components and shrinks negligible carry-over effects toward zero, thereby enabling their practical identification. Through simulation studies and application to real data, the methodology demonstrates improved estimation accuracy when complex carry-over effects are present, outperforming models that assume simple or no carry-over. This work represents the first rigorous and generalizable approach for modeling complex carry-over effects in repeated-measures crossover designs.