ISA3: a 3-dimensional expansion of instance space analysis

Abstract The experimental validation of algorithms depends strongly on the characteristics of the test set used. Ideally, such a set should exhibit diverse characteristics that challenge an algorithm and are present in real-world problems. An approach to examine the diversity and representativeness of a test set is Instance Space Analysis, which uses a 2D projection to visualise the test set while identifying the strengths and weaknesses of competing algorithms. However, this has the limitation of discarding potentially useful information while crowding out the space as more features and algorithms are considered. This paper describes an extension of Instance Space Analysis into 3D, which retains a higher degree of information while maintaining the explainability of the visualisation by finding the rotations that maximise the linear trends. In addition to the expansion to 3D, a new algorithm for identifying portfolio footprints is introduced, offering a more robust and reliable method for identifying footprints in both 2D and 3D instance spaces. As a case study, we present a performance analysis of unsupervised anomaly detection methods, subject to changes in the normalisation technique. The results demonstrate the advantages by identifying regions of strength of algorithms previously thought to be underpowered.