Evaluation of machine learning methods in medicine: real data application

Objective: One of the aims of a health study is to identify risk factors associated with the disease or to obtain predictive models for classification such as healthy / diseased. When the aim of a health study is classification, machine learning methods are widely used. The aim of this study was to evaluate the performance of the machine learning method, for different sample size, prevalence and determination coefficient in real data sets.Method: The data were randomly split into 70% training and 30% test set, and Logistic regression, Decision tree, Random Forest, Support Vector Machine, and Naive Bayes were applied to the training set. The performance measure (Accuracy, Area Under Curve and Adjusted F Measure) of the methods in the test set were saved. These procedures were performed in the R 3.5 1.Results: When all variables in the data are categorical, and R2 is low with a moderate sample size, the Naive Bayes (NB) method exhibited higher performance. When all variables in the data are continuous, and R2 is moderate with a low sample size, support vector machines (SVM) method demonstrated superior performance. In cases where the dataset has a high number of categorical variables and a high R2, the Naive Bayes (NB) method outperformed others. The Random Forest (RF) method showed higher performance when R2 is high, and the sample size is moderate.Conclusion: This study provides valuable insights for researchers dealing with classification problems, guiding them to choose the most effective machine learning based on the characteristics of the datasets.