Clustering High-Dimensional Spaces Using a Modified EM Algorithm with Fractional Order Assignment

Clustering high-dimensional data presents significant challenges due to the curse of dimensionality, which complicates the detection of meaningful patterns and clusters. Traditional methods struggle to manage the increasing complexity, often becoming sensitive to noise, inefficient in capturing intricate relationships, and unable to handle uncertainties effectively. To address these issues, this study introduces EM-FASL, a novel clustering algorithm designed to improve accuracy in high-dimensional spaces by accounting for complex relationships and uncertainties. EM-FASL is an adaptation of the Expectation-Maximization (EM) algorithm, enhanced through fractional order assignment and logistic regression (LR). The method begins by initializing the parameters for a Gaussian Mixture Model (GMM) and an LR model. In the Expectation (E) step, it fits the GMM, computes fractional assignments, and evaluates fractional entropy to measure the uncertainty in assignments. In the Maximization (M) step, the GMM parameters are updated using the fractional assignments, while LR coefficients adjust the mean vectors of the GMM. The algorithm iterates between these steps until fractional entropy converges. The algorithm is evaluated on a range of high-dimensional datasets from the UCI repository, demonstrating its robustness across diverse data structures. Experimental results demonstrate that EM-FASL significantly outperforms existing clustering methods, offering superior accuracy in high-dimensional data by effectively capturing intricate data patterns. This innovative approach provides a robust solution to the challenges of high-dimensional clustering and opens up new avenues for future research.