Design and Validation of a Governed Data Lakehouse Architecture for Big Data Analytics: Evidence from a Smart Port
Big Data platforms are expected to unify data engineering, governance and machine learning in one environment, and the data lakehouse has become the dominant architectural answer. Empirical evaluations on real operational data remain scarce, and those that exist tend to validate a downstream model rather than the architecture itself. This study designs, implements and evaluates a governed data lakehouse following Design Science Research. The artefact organises Apache Spark, Delta Lake, Unity Catalog and MLflow into a medallion architecture, and is exercised on operational data from the Port of Sines: 36,773 reporting events yielding 8148 berth visits by 1923 vessels between October 2020 and September 2024. The architecture is evaluated against explicit criteria rather than by proxy. Ingestion throughput stays between 16.8 and 18.5 thousand records per second as volume quadruples; the curated Gold layer answers analytical queries 100 times faster than recomputation from source and compacts storage 6.6-fold; an explicit quality ledger and a physical-bound quarantine make governance auditable and measurably beneficial downstream, restoring a correlation structure that 55 out-of-bound values had erased; and an incremental refresh path updates the analytical layer in 0.98 ms at the weekly cadence, at a cost that does not grow with accumulated history. A berth-time regression under rolling-origin validation, benchmarked against tuned ensembles, operational heuristics and time-aware sequence models, serves as the analytical probe rather than as the contribution. The study contributes a measured, replicable reference architecture for governed Big Data analytics.
Authors
- Maria da Graça Costa (ORCID: https://orcid.org/0000-0003-1324-2685)
- Tiago Pinho (ORCID: https://orcid.org/0000-0003-2546-5392)
- Ana Mendes (ORCID: https://orcid.org/0000-0003-3229-4712)
- Marcela Castro
- Klysman Rezende Alves Vieira
Institutions
- University of Beira Interior (PT)
- Life Quality Research Centre (PT)
- Universidade Politécnica de Setúbal (PT)
- Universidade Nova de Lisboa (PT)
Publication Details
- Journal
- Big Data and Cognitive Computing
- Published
- 2026-09-28
- DOI
- https://doi.org/10.3390/bdcc10100331
- Primary Topic
- Big Data and Business Intelligence
- Type
- article
- Field-Weighted Citation Impact
- 0.00