Inference for generalized additive mixed models via penalized marginal likelihood

The Laplace approximation is sometimes not sufficiently accurate for smoothing parameter estimation in generalized additive mixed models. A novel estimation strategy is proposed that solves this problem and leads to estimates exhibiting the correct statistical properties. • First accurate approximation to the likelihood in models with both random effects and penalized splines. • Two-stage approach to smoothing parameter and variance component estimation. • Application to Bernoulli generalized additive mixed models.