Design and Validation of a Governed Data Lakehouse Architecture for Big Data Analytics: Evidence from a Smart Port

Big Data platforms are expected to unify data engineering, governance and machine learning in one environment, and the data lakehouse has become the dominant architectural answer. Empirical evaluations on real operational data remain scarce, and those that exist tend to validate a downstream model rather than the architecture itself. This study designs, implements and evaluates a governed data lakehouse following Design Science Research. The artefact organises Apache Spark, Delta Lake, Unity Catalog and MLflow into a medallion architecture, and is exercised on operational data from the Port of Sines: 36,773 reporting events yielding 8148 berth visits by 1923 vessels between October 2020 and September 2024. The architecture is evaluated against explicit criteria rather than by proxy. Ingestion throughput stays between 16.8 and 18.5 thousand records per second as volume quadruples; the curated Gold layer answers analytical queries 100 times faster than recomputation from source and compacts storage 6.6-fold; an explicit quality ledger and a physical-bound quarantine make governance auditable and measurably beneficial downstream, restoring a correlation structure that 55 out-of-bound values had erased; and an incremental refresh path updates the analytical layer in 0.98 ms at the weekly cadence, at a cost that does not grow with accumulated history. A berth-time regression under rolling-origin validation, benchmarked against tuned ensembles, operational heuristics and time-aware sequence models, serves as the analytical probe rather than as the contribution. The study contributes a measured, replicable reference architecture for governed Big Data analytics.

Authors

Institutions

Publication Details

Journal
Big Data and Cognitive Computing
Published
2026-09-28
DOI
https://doi.org/10.3390/bdcc10100331
Primary Topic
Big Data and Business Intelligence
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
article

Design and Validation of a Governed Data Lakehouse Architecture for Big Data Analytics: Evidence from a Smart Port

Maria da Graça Costa, Tiago Pinho, Ana Mendes, Marcela Castro et al.
Big Data and Cognitive Computing
Big Data and Business Intelligence
article

Design and Validation of a Governed Data Lakehouse Architecture for Big Data Analytics: Evidence from a Smart Port

Maria da Graça Costa, Tiago Pinho, Ana Mendes, Marcela Castro, Klysman Rezende Alves Vieira
article en

Abstract

Big Data platforms are expected to unify data engineering, governance and machine learning in one environment, and the data lakehouse has become the dominant architectural answer. Empirical evaluations on real operational data remain scarce, and those that exist tend to validate a downstream model rather than the architecture itself. This study designs, implements and evaluates a governed data lakehouse following Design Science Research. The artefact organises Apache Spark, Delta Lake, Unity Catalog and MLflow into a medallion architecture, and is exercised on operational data from the Port of Sines: 36,773 reporting events yielding 8148 berth visits by 1923 vessels between October 2020 and September 2024. The architecture is evaluated against explicit criteria rather than by proxy. Ingestion throughput stays between 16.8 and 18.5 thousand records per second as volume quadruples; the curated Gold layer answers analytical queries 100 times faster than recomputation from source and compacts storage 6.6-fold; an explicit quality ledger and a physical-bound quarantine make governance auditable and measurably beneficial downstream, restoring a correlation structure that 55 out-of-bound values had erased; and an incremental refresh path updates the analytical layer in 0.98 ms at the weekly cadence, at a cost that does not grow with accumulated history. A berth-time regression under rolling-origin validation, benchmarked against tuned ensembles, operational heuristics and time-aware sequence models, serves as the analytical probe rather than as the contribution. The study contributes a measured, replicable reference architecture for governed Big Data analytics.

Big Data and Cognitive ComputingVol. 10(10)
University of Beira Interior (PT), Life Quality Research Centre (PT), Universidade Politécnica de Setúbal (PT), Universidade Nova de Lisboa (PT)
Industry, innovation and infrastructure
Openalex Percentile: Top 6%
Big Data and Business Intelligence
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.