Addressing K-Nn Limitations Through Boosted Multi-Algorithm Nearest Neighbour Ensembles

Determining the degree of similarity between data points accurately is a fundamental challenge for many classification problems, particularly in high-dimensional spaces. K-nearest neighbors, or k-NN, is a basic and popular technique because it is straightforward, easy to understand, and works well with small datasets. On large, high-dimensional datasets, however, traditional k-NN methods suffer from severe limitations in terms of runtime performance and classification quality. Poor generalization results from these restrictions, which are made worse in cases of imbalanced classes when the target neighbors might not fall inside restricted local spheres. In order to address the inherent drawbacks of k-NN, this paper proposes a novel solution: ensembling multiple alternative nearest neighbor search algorithms. In particular, we present the Omni-kNN framework that combines sophisticated approximate nearest neighbor (ANN) algorithms like Annoy, ball trees, and Hierarchical Navigable Small World (HNSW) graphs with multi-step k-NN. Our framework aims to maintain computational efficiency while improving classification accuracy and robustness by utilizing the advantages of these various approaches.To further encourage more diversity during meta-classifier training and lower the correlation between individual learners, we also include highly randomized trees. Extensive evaluations of this heterogeneous ensemble framework are conducted on a variety of real-world datasets, including ones with substantial class imbalance and size. Our findings show significant gains in performance metrics, highlighting the Omni-kNN framework's potential to tackle challenging classification problems in modern machine learning applications.