Using logical data warehouse in the process of big data integration and big data analytics in organisational sector

This thesis first reviews different aspects of big data management, including data integrationand traditional data warehouse, and their associated challenges. In many enterprises today,strategic information plays an increasingly important role in helping decision makers reach orjustify decisions about their business. An effective decision-making process benefits from datathat is frequently extracted from several heterogenous data sources. Big data developmentaffects enterprises across various sectors [81]. The data extracted are from various databaseswhich are from different organisation systems and this has resulted in organisation managersdealing with issues of integrating several databases from independent data sources withdifferent semantic or no canonical concept description. The increase of data volume, high speedof data generation and the increasing growth of different data from heterogeneous sources haveled to data warehouse implementations that are relatively unresponsive to real-time changes inthe business environment. Data warehouses are most valuable assets for enterprises which dealwith a lot of business information. Given the volume of data dealt with, consuming too manyresources can be expensive to acquire and maintain. Data warehouses mainly store historicaland current business data for decision making. Data integration in today’s world is more thanjust integration, the need for data integration is now to offer reusable data integration solutionsand a complete innovative portfolio of data capabilities in a fast-moving ecosystem [83].Problems included in big data integration are the increased duplications of data especiallyunwanted or unused data and has resulted in data redundancy, problem like data accessibilityto regulatory bodies or agencies for finding out trends or statistics of a particular challengefaced in a city to carry out proper decision making, cost of data transformation and movementfrom heterogeneous sources into a central database, especially in the big data environment.This research addresses the problem of conflicting data formats. This thesis employed the useof questionnaire (interview questions) which were sent out to different public organisations.Responses from staff of several public organisations were used to compare and formulatedifferent scenarios and construct suitable case study. The use of the questionnaire gave theresearcher the ability to see and identify the common challenges in their various organisations,compared their challenges with their current data integration management tools, studied eachof their business requirements and drew conclusions as to whether Data Virtualisation will bethe best solution to their business requirements. At the end of the qualitative approach andliterature review, the research was able to portray that ETL techniques is widely used, and thereis little or no knowledge about Logical data warehouse as an alternative approach to anorganisation. The research was also able to elaborate the difference between the traditional datawarehouse and logical data warehouse using the demonstration of prototype, showing theadvantages of data virtualization over ETL techniques. The prototype also reveals theimportance of SLAs construction and SLAs’ requirement involved in the achievement of theframework. This research also finds out that Service Level Agreement (SLA) can be a solutionto some of the v’s of big data challenge which are the volume and variety and realised thatthere may be cost in maintaining SLAs between two parties.