Kernel density estimation in metric spaces

Abstract Non‐Euclidean data analysis has become a crucial task in modern statistics, given the rapid emergence of non‐Euclidean data in various fields. However, fundamental tools for non‐Euclidean statistics are still lacking or under development. In this paper, we propose a generalized probability density estimation method for metric spaces, based on the metric distribution function. We extend the conventional kernel density estimation method to metric spaces and introduce local and global versions of metric kernel density estimation. We establish their large sample properties under regularity conditions. Furthermore, we develop a mean integrated squared error‐based bandwidth selection criterion for these new estimators. Extensive simulations under various settings are conducted to demonstrate the finite‐sample performance of our proposed estimators. We exemplify the efficacy of our methods using hippocampal data, specifically capturing hippocampal surface changes of representative samples at different levels of Alzheimer's disease (AD) and exploring factors affecting hippocampus shape and AD severity.