A hybrid meta-heuristic optimisation and unsupervised learning framework for predictive stability enhancement of single-machine infinite-bus power systems

Accelerating renewable penetration into modern interconnected grids has amplified low-frequency electromechanical oscillations, eroding the dynamic stability margins of synchronous generators. A unified two-phase framework is proposed to address Power System Stabiliser (PSS) parameter optimisation and real-time stability estimation for the Single-Machine Infinite-Bus (SMIB) benchmark. In Phase I, three nature-inspired meta-heuristic algorithms, namely Ant Colony Optimisation (ACO), Harris Hawks Optimisation (HHO), and Passing Thought-Based Optimisation (PTBO), tune the PSS lead-lag parameters $$\{K_{\textrm{PSS}},T_{1},T_{2}\}$$ by minimising the Integral of Time-weighted Absolute Error (ITAE) of rotor speed deviation across a 50 MVA SMIB benchmark. MATLAB/Simulink simulations over three disturbance severities confirm that HHO achieves the lowest settling time (3.5 s), while PTBO attains the lowest ITAE cost observed in these experiments (1514.5) within 10 iterations. In Phase II, the optimised SMIB dynamic-response dataset trains and cross-evaluates three unsupervised machine learning (ML) models, K-Means clustering, Gaussian Mixture Model (GMM), and Self-Organising Maps (SOM), for real-time PSS parameter estimation under unseen operating conditions. Evaluation by Silhouette Score, Davies-Bouldin Index (DBI), and Calinski-Harabasz (CH) Score ranks GMM best across all three metrics (Silhouette $$=$$ 0.2485; DBI $$=$$ 1.458; CH $$=$$ 393.68), confirming its superiority in representing the non-spherical, correlated structure of the disturbance feature space. On closed-loop performance, however, SOM is the strongest model, giving the shortest settling time in two of the three disturbance cases and the largest mean settling-time reduction at 23.3%; SOM is accordingly identified as the best overall model for stabiliser parameter estimation, and GMM as the best representation of the feature space. Quantitatively, settling time falls from 9.5 s uncontrolled to 3.5 s, a reduction of 63.2%, and from 5.2 s under conventional tuning, a reduction of 32.7%; peak speed excursion falls from $$+1.68$$ to $$+1.20$$ relative to the conventional stabiliser, a reduction of 28.6%; the lowest observed ITAE is 1514.5 against 1519.0 and 1521.0 for the two competing optimisers; parameter estimation reduces settling time by 17.3%, 22.1%, and 23.3% on average for K-Means, GMM, and SOM respectively relative to a fixed nominal stabiliser; and estimation completes in approximately 0.3 ms against 30–45 s for a full re-optimisation. The framework is demonstrated on a single-machine infinite-bus benchmark under offline simulation, and the timing figure establishes computational feasibility for a real-time implementation rather than real-time capability itself.