Challenges in Caryophanales species delimitation: comparative evaluation of genomic similarity metrics and phylogenomics based on six environmental strains

Species delimitation within Caryophanales remains challenging due to extensive polyphyly, particularly within the Bacillus subtilis and Bacillus cereus sensu lato groups. We compared multiple genome-based approaches – ANI (FastANI, skani), dDDH, TZMD, and single-copy gene (SCG) phylogenomics – for species assignment of six environmental strains from post-maize cultivation soils, performing comprehensive pairwise comparisons among all available RefSeq genomes at complete and chromosome assembly levels for each taxonomic neighbourhood. Results revealed highly variable species boundaries. Two strains showed unambiguous assignment to B. subtilis and B. licheniformis , with all ANI comparisons exceeding the 95% threshold and clear phylogenomic separation. In contrast, strain Bac2 belonged to the Operational Group Bacillus amyloliquefaciens , where between-species ANI exceeded within- B. amyloliquefaciens values and SCG phylogenomics revealed extensive species intermixing, demonstrating fundamental boundary inconsistency. Strain zielonkawy was assigned to B. cereus sensu stricto genomospecies, with extensive B. cereus s.s. – B. thuringiensis intermixing characteristic of this complex. Two additional strains were assigned to Priestia megaterium and Paenibacillus amylolyticus ; however, both ANI multi-comparison analysis and SCG phylogenomics suggested possible species intermixing or database mislabelling, though the Paenibacillus dataset was limited by small sample size. To our knowledge, this is the first study to quantitatively document species delimitation problems between Pr. megaterium and Pr. aryabhattai , and between P. amylolyticus and P. xylanexedens . Quantitative concordance analysis revealed a non-linear relationship between FastANI and dDDH formula 2 (quadratic R 2 = 0.991), with the 70% dDDH threshold corresponding to 96.16% ANI rather than the conventionally assumed 95%, confirming that dDDH formula 2 is the more conservative metric. Concordance between skani genomic distances and SCG phylogenomic patristic distances was high across all datasets (R 2 = 0.900–0.997), confirming overall methodological agreement; Spearman rank correlation was also high, but consistently lower (ρ = 0.780–0.916), indicating that the precise rank order of closest neighbours can differ between methods even when overall genomic structure is concordant. We propose within- and between-species ANI multi-comparison analysis with PERMANOVA and post-hoc pairwise testing as a complementary diagnostic tool to SCG phylogenomics for evaluating species boundary validity in taxonomically complex groups.