Easy and Hard Partitions: An Investigation On Dataset Partitions in Biomedical Image Analysis

Researchers and practitioners are often provided with bench-marking datasets that include predefined training and testing partitions. Their task is to design and optimize machine learning models using the training data, aiming to achieve high performance (e.g., accuracy) on the test data. While mod-ern advancements in deep learning-such as transformers and models like Mamba-have made model research increasingly sophisticated, the fixed dataset partition setup raises an im-portant question: Can dataset construction, as the foundation of model development, be improved to better support specific goals in model benchmarking and evaluation? In this paper, we propose a new approach, Easy-and-Hard Partitions (EHP), for constructing benchmarking datasets to advance the development of biomedical AI models. Given a set of labeled samples, our aim is to partition it into two sub-sets (training and testing) with specific objectives for model evaluation. Our main interests lie in two goals: (1) evaluating a model when minimal data shift exists between the training and test sets, achieved by creating a partition where the training set is highly representative of the test set; and (2) as-sessing a model's performance under a more pronounced data shift, achieved by ensuring a significant difference between the training and test sets. We demonstrate the use of these partitions through experiments on classification and segmen-tation tasks in biomedical image analysis. EHP provides a structured approach to dataset partitioning, facilitating model assessment and development in biomedical applications.