Reproducibility and Research Contribution in AI

Reproducibility has emerged as a central issue in artificial intelligence research today because many reported results are hard to reproduce, because of lack of experimental documentation and variation in implementations. This paper performs a reproducibility analysis of the study proposed in Learning Geometry-Aware Representations for New Intent Discovery, in which the GeoID framework for discovering previously-unseen intent categories in conversational datasets is introduced. The GeoID approach is a combination of sentence encoding algorithm using transformers and geometry-aware supervisor using a simplex Equiangular Tight Frame (ETF) classifier and Second pseudo-labeling strategy based on optimal transport and clustering signals.To assess the results of the reported research, the experiments are reproduced in this study using the official GitHub (https://github.com/zjutangk/GeoID ) implementation and benchmark datasets, BANKING77, StackOverflow and CLINC150. The obtained results are compared with the results obtained in the original paper using clustering measures such as normalized mutual information (NMI), adjusted Rand index (ARI), and clustering accuracy (ACC). The results of the replicate experiments have similar performance trends to the original work, indicating that the GeoID framework has good reproducibility under similar experimental conditions. However, minor differences in performance and limited performance due to implementation issues demonstrate the need for transparent reporting of experiments and good documentation of code. The study further identifies opportunities for improving the GeoID framework in the areas of configuration management, better seed control and evaluation with unknown cluster count.