Toward Measuring and Understanding the Overvalidation Phenomena

Over the past decade, advances in Machine Learning have significantly expanded both the number and complexity of algorithms. Consequently, solving specific tasks now requires making numerous choices, such as selecting the appropriate algorithm, architecture, and corresponding hyper-parameters. Although various methods have been proposed to expedite this search process, the final selection is typically made using a holdout set. While this practical approach is widely accepted, it can lead to the issue of overvalidation when the number of choices is large. Overvalidation refers to the bias in holdout performance, which can result in the incorrect selection of the optimal choice so to be not aligned with the technical needs. This issue can be mitigated by improving the quantity and quality of data in the validation set or through resampling, but it reappears as the number of choices increases. Thus, the challenge of better understanding and detecting overvalidation to avoid or further mitigate it remains unresolved. In this paper, we address this problem by measuring and understanding the overvalidation phenomenon using statistical learning theory and testing it with real-world examples.