An Algorithmic Approach to Generate A Dataset for Analyzing Diagrammatic Reasoning Ability Through Machine Learning

Raven Progressive Matrices (RPM) is a well-known and commonly used non-verbal intelligence test. In this paper, we present a novel Diagrammatic Reasoning problem-related dataset generated algorithmically. Using the proposed algorithm, we generated 1000 RPM problems designed to evaluate machine learning models’ ability to solve visual reasoning tasks. This dataset (RPMEd, RPM for education) includes variations of geometric shapes like ellipses, rectangles, arrows, hexagons, pentagons, and others, with applied transformations such as rotation, scaling, and flipping the shape’s fill. Each matrix in the dataset has a corresponding correct answer stored in a text file for supervised learning tasks. This dataset is highly suitable for training and benchmarking machine learning models, like convolutional neural networks and vision transformers, for visual reasoning tasks. The dataset created through this method provides a rich resource for evaluating the problem-solving abilities of both humans and artificial intelligence systems, contributing to advancements in the field of non-verbal reasoning and intelligence measurement. We experimented with five transformer-based models on our dataset and found that the test accuracy of the model based on a Data-efficient Image Transformer(DeiT) outperforms other well-known transformer-based models. The test accuracy of DeiT is coming out to be 74%, which may be improved in future work. Our findings indicate that contemporary Transformer-based models exhibit suboptimal performance on RPMEd, highlighting the necessity for more robust, specialized models to address RPM challenges. We suggest RPMEd as a benchmark for assessing the efficacy of current transformer-based models in addressing challenges and encourage the scientific community to collaborate in developing more effective solutions.