The challenges of non-identifiability and a penalized maximum likelihood estimator for the beta mixture model

This thesis explores statistical inference for the finite mixture models, with a particular focus on beta mixture models, which are widely used in biostatistics, bioinformatics, and computer science. It addresses significant issues such as unbounded likelihood and non-identifiability, which can complicate parameter estimation. To overcome the obstacle caused by the unbounded likelihood, we propose a penalized maximum likelihood estimation approach by adding a penalty term to the log-likelihood function, leading to stable parameter estimation. Additionally, we derive a closed-form expression for testing non-identifiability in beta mixture models. The effectiveness of our penalized approach is evaluated through simulation studies and compared with alternative approaches, such as the method of moments. Practical applicability is demonstrated through applications to DNA methylation analysis and local false discovery rate estimation. Finally, we suggest several directions for future research.