Bulid a Data-pipeline to Process & Analyze Data

This project focuses on building a robust data pipeline for processing movie and TV show data stored as CSV files on GitHub. The data is first ingested into Azure Data Lake Storage, forming the base for further processing. Using Databricks Auto Loader, the pipeline supports incremental data loading, allowing it to handle new data efficiently. The pipeline adopts a multi-layered architecture: Bronze Layer: Stores raw data. Silver Layer: Contains cleaned and transformed data. Gold Layer: Hosts refined datasets ready for reporting and analytics. Delta Live Tables (DLT) are used in the Gold layer to ensure streamlined data management and maintain data quality throughout the pipeline.