Automated Data Analyser using Generative AI

In the era of digital transformation, data analysis plays a significant role in data science and machine learning, it act as a decision support system. Extracting insights from data is really difficult and effort-intensive activity. The Automated data Analyser is developed for such users who need automated data insights and machine learning model recommendations. It is built using python framework streamlit and uses google Gemini for generating the insights and suggesting machine learning algorithms for the given dataset. Initially the dataset is given as the only input nothing other input is needed as the system is automated it automatically pre-processes the data and generates the insightful data visualizations along with the descriptions. If the dataset is huge and contains more rows then the system uses dimensionality reduction techniques such as Principal Component Analysis (PCA) to reduce the size of data and preserve the insights. By reducing the size of dataset using dimensionality reduction techniques it takes less time to analyse the dataset. It is an efficient system for huge data sets. The system is also capable to show the mathematical analysis of data to analyse the spread of data. To produce the data insights by Gemini LLM a small set of data along with prompt is given to LLM to generate the insights and recommendations. The application uses SQLite database to store the visualizations and insights. Finally the report is generated which includes the data insights, data visualizations and machine learning model suggestions. The combination of Generative AI, Statistical Analysis and ML Automation is a great fit in the world of data analytics. This research work helps automate the Exploratory Data Analysis (EDA) tasks.