A Feature Selection Approach Based on Information Theory with Application to the International Monetary Fund and World Bank Economic Datasets

Feature selection is a type of dimension reduction problem that is usually used as a pre-processing stage before starting to analyze a set of data. In the literature on feature selection approaches, factors such as nonlinearity, relevancy, redundancy concepts among the features, and also class variables have been considered individually or partially together in order to remove the unimportant features. In this study, for the first time, an information theoretic-based evaluation measure is applied for feature selection purposes and a unified approach is developed considering the above-mentioned factors for selecting the most important features. This unified measure, for the first time attempts to simultaneously observe the effect of all the factors mentioned above and make a tradeoff between them to diminish the effect of neglecting some of these players that are used in distinguishing the importance of features. So, the proposed approach has the advantage of being able to work with both supervised and unsupervised types of data, considers the linear and nonlinear relationships among the features, and attempts to take care of the pairwise correlation among the features instead of considering a general behavior of them. The proposed approach is applied to some economic datasets from the two most popular databases which are the International Monetary Fund database (IMF data) and World Bank Metadata. According to the obtained results, some comparisons with the methods of the literature are done and the sensitivity of the proposed approach is studied over its parameters.