Addressing threats to validity in large dataset research

Large dataset research offers new opportunities across biomedicine, but it also faces major threats to validity.These threats are underappreciated and often not recognized or managed appropriately.We dissect here 4 major types of validity threats: (1) P-value inflation and trivial effects; (2) residual and differential confounding across multiple sites; (3) issues surrounding harmonization and measurement error; and (4) causal identification in the presence of rich covariates.For each of them we provide their theoretical background, empirical evidence, and simulations that showcase how they operate.We make recommendations that may offer guidance for analyses and inferences from large datasets across biomedicine.These include: deciding on the placement in a continuum of exploration versus targeted causation; evaluating both statistical significance and effect sizes; modeling site heterogeneity; treating harmonization as measurement modeling; considering causal diagrams to guide adjustment; checking covariate balance; embedding sensitivity analysis and exploration of robustness; reporting limitations structurally; considering falsification calibration of performance; considering multi-analyst approaches; enhancing sharing, openness, transparency, and documentation; and considering encompassing analyses rather than least publishable units.Improving large dataset research and adequately addressing these threats is essential for the integrity of the scientific record and eventually also for informing policy and clinical practice.