Efficient and Secure Transformation of Log Files using ADLS GEN2 Architecture

Web server log encompasses a document where all the activities transpiring on a web server are carefully recorded. It captures each request made to the server like request type, IP address of the device, what files where requested, date & time of request, name and location of requested file, file size etc. These files are stored in their raw form for a long time for analysis and taking high quality decisions which causes storage problem and security threat as data present in the log files are in raw format. This paper focuses on providing an economical, secure and high - performance solution for the storage of large amount of raw log files. The proposed system includes Azure Data Lake Storage Gen2 which allows large volumes of data to be stored in their raw form as well as they are subjected to transformation and advanced analysis processes without the need of a structured writing scheme. This paper mainly provides solution that is affordable and more accessible to perform web server log data ingestion, storage and transformation over Data Lake. This paper also proposes the use of Azure Trigger Function that transforms the log files into parquet files which reduces the storage space compared to their original size. A hierarchical data storage model has also been proposed for shared access to data over different layers in the Data Lake architecture, on top of which Data Lifecycle Management rules have been proposed for storage cost efficiency. The aim is to maintain this data in the long term to be used in future advanced analytics processes by cross - referencing with other organizational or external data which could bring important benefits.