Proposed Metadata Management Model for Data Lakehouse

The exponential growth in the amount and diversity of data organizations generate has presented significant challenges in data storage, processing, and management. Data Lakehouses, a hybrid solution that combines the capabilities of Data Warehouses and Data Lakes, have been developed to address these challenges. However, metadata management in these environments is crucial to prevent data from becoming inaccessible and unwieldy. We propose a metadata management model for a Data Lakehouse based on a functional architecture that integrates functional metadata features for a Data Lake with business and maintenance metadata for a Data Warehouse. We include the results of an initial implementation and discuss its impact on data accessibility and governance.

Authors

Institutions

Publication Details

Published
2024-10-28
DOI
https://doi.org/10.1109/sccc63879.2024.10767658
Primary Topic
Data Quality and Management
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Proposed Metadata Management Model for Data Lakehouse

Ania Cravero, Pablo Rain
Data Quality and Management
article

Proposed Metadata Management Model for Data Lakehouse

Ania Cravero, Pablo Rain
article en

Abstract

The exponential growth in the amount and diversity of data organizations generate has presented significant challenges in data storage, processing, and management. Data Lakehouses, a hybrid solution that combines the capabilities of Data Warehouses and Data Lakes, have been developed to address these challenges. However, metadata management in these environments is crucial to prevent data from becoming inaccessible and unwieldy. We propose a metadata management model for a Data Lakehouse based on a functional architecture that integrates functional metadata features for a Data Lake with business and maintenance metadata for a Data Warehouse. We include the results of an initial implementation and discuss its impact on data accessibility and governance.

Universidad de La Frontera (CL)
Climate action
Openalex Percentile: Top 73%
Data Quality and Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.