Improving Data Reliability Assessment in ETL Processes through Quality Scoring Technique in Data Analytics

The foundation of a relevant and accurate data analysis is reliable data. Technique and measurement are essential to evaluate current data quality regarding reliability and establish a baseline for ongoing improvement initiatives. Without tools or visualizations, data engineers may find it challenging to monitor and maintain the reliability of the massive data from the extraction, transformation, and loading (ETL) data load process. Data reliability assessment is a helpful technique in analyzing the quality of data reliability and information on the present state of data before commencing any analytics. The proposed technique hinges on the metric and measurement defining data reliability and the dashboard platform where the integration with the user in dictating the weight of data and the final output, which is the final data reliability score, will be projected. The score obtained affirms whether improvements are needed on the data or if an organization can proceed with data analytics. The technique considers the data extraction, transformation, and loading (ETL) procedures used to gather datasets. Data significance or weight was determined according to the analytics needs and preferences, indicating an acceptable score for generating insights. Ultimately, when utilizing the data reliability assessment metrics technique, we are credited with an overall picture of our data’s reliability aspect, as only one look is offered based on the intended analysis. This new approach boosts the confidence among data practitioners and stakeholders, especially those relying on findings generated from data analysis. Furthermore, the overview assists in enhancing the current state of data, where the derived score helps identify possible areas of improvement in the ETL process. Accuracy and efficiency assessment of the proposed technique also showed positive feedback in measuring the method in measuring the reliability of data.