Efficient Estimation For Semiparametric Regression Models With Big Data

Semiparametric models have been widely used in biomedical studies, econometrics, and social sciences. However, the computational challenges of fitting semiparametric models restrict their use to small datasets with hundreds or thousands of observations. For datasets involving millions of subjects, current methods are often impractical, highlighting the need for more computationally efficient approaches.In the first project, we propose a computationally efficient method for fitting the Cox proportional hazards model to large datasets. Specifically, we perform maximum partial likelihood estimation on a small subset of data and improve the initial estimator by incorporating the remaining data through one-step estimation with estimated efficient score functions. We show that the final estimator has the same asymptotic distribution as the conventional maximum partial likelihood estimator using the whole dataset but requires a small fraction of computation time. We demonstrate the usefulness of the proposed method through extensive simulation studies and an application to the UK Biobank data.The second project generalizes this approach to a broader class of semiparametric problems. We calculate an initial estimator for the finite-dimensional parameters using a small fraction of data and improve its efficiency by one-step estimation with semiparametric efficient score functions on the remaining data. We estimate the efficient scores by the numerical differences of the profile log-likelihood function, eliminating the need for an analytic form of these scores. The resulting estimator is consistent, asymptotically normal, and semiparametric efficient. The proposed approach substantially reduces the computational burden compared to standard methods. We validate the proposed methods through simulations on fitting the Cox model to interval-censored failure times and the partially linear model to continuous outcomes, and apply it to interval-censored data from the UK Biobank.The third project focuses on proportional hazards and intensity models with random effects in large datasets. We perform nonparametric maximum likelihood estimation on a small subset of data and improve the initial estimator by one-step estimation with estimated efficient scores on the remaining data. The final estimator is consistent, asymptotically normal, and semiparametric efficient. We illustrate the method’s efficiency through comprehensive simulations and a UK Biobank application.