Agent Based Framework For Automated Data Cleaning And Validation

Real-world datasets can contain missing values, duplicate records, inconsistent data, outliers, and invalid entries. Manual cleaning is time-consuming, and the result must be checked to determine whether it satisfies data-quality requirements. This paper presents an agent-based framework for automated data cleaning, validation, and merging implemented using Python, NumPy, and Pandas, with Flask and scikit-learn used in the prototype application. Different agents handle schema profiling, issue detection, reasoning, cleaning, and validation. The system detects candidate problems using type inference, pattern matching, hash matching, and IQR-based outlier detection. A Reasoning Agent proposes possible cleaning actions, which are presented to the user for approval or rejection before accepted transformations are applied. The Validation Agent then checks the output against configured quality rules. If validation identifies remaining problems, the issue can be returned to the reasoning stage for a further proposed correction and user decision. The framework also supports merging two datasets using schema matching. Screenshots from a customer-data demonstration show a reduction in reported missing values and duplicate rows, while the validation dashboard still reports failed checks. These observations describe prototype interface output rather than a controlled benchmark; they do not establish general cleaning accuracy or performance. The framework aims to reduce repetitive manual work while retaining human oversight and an explicit validation step before data are used for analysis or machine learning.