A Preprocessing Algorithm to Identify Errors in Datasets for AutoML

Machine learning algorithms need a dataset for training and performance evaluation. Usually, the dataset samples collected are incomplete in many aspects. There may be an incorrect recording of one or more features values, an unknown value (empty), an incorrect symbol for categorical features, etc. The dataset must be corrected (preprocessed) before presenting to the machine learning algorithm for training and performance evaluation. But datasets usually consist of a large number of samples and hence manually searching for defects is time-consuming and difficult if not impossible. This paper presents an algorithm to preprocess datasets to identify and list errors in the datasets. The effort is in the direction of automated machine learning (autoML) pipeline development to ease the use of ML by non-experts. The algorithm is tested with a dataset that has all kinds of data types and errors in the dataset are listed.