IntelStack: An Open-Source, Sovereign and Resilient Data-Science Platform for Resource-Constrained Research Environments - Preprint

Small and medium-sized organisations that need advanced data-science capability rarely have the budget, staff or legal capacity to adopt large commercial cloud platforms in a sovereign way. At the same time, single-purpose open-source tools deployed in isolation do not by themselves constitute a usable, governed data-science environment. This paper presents IntelStack, a sovereign and resilient platform that integrates multiple concurrently running applications on a single virtual machine with internal integration and unified governance: fourteen mature, carefully selected open-source projects (Apache Airflow, LinkedIn DataHub, DHIS2, Project Jupyter's JupyterHub, Keycloak, MinIO, NGINX, ODK Central, Ollama, Open WebUI, pgAdmin4, PostgreSQL, SNOMED International's Snowstorm, and Apache Superset) and five custom-built components (GlobalSecurity, GlobalStatus, IntelWeb, RWKV and SnowstormExtensionsAndAI) that supply identity-aware entry-point security, unified observability, Open-Source Intelligence (OSINT) capability, AI-assisted information analysis, public-facing dissemination, sovereign large-language-model usage and AI-assisted terminology extensions. The platform was developed and first operationally tested in the context of the research project "Interoperability and utilisation of epidemiological surveillance data with artificial intelligence", which required the rapid development of a low-cost, standalone/sovereign and resilient data-science environment for public-health data experimentation. The architecture, however, is domain-agnostic and is reported here as a general-purpose sovereign data-science platform rather than as a public-health-specific tool. We describe the platform's layered architecture (acquisition, orchestration, storage, cataloguing, semantic enrichment, prediction/model execution, analytics, dissemination, identity, entry-point security and observability), map each of the thirteen stages of a generic data-science lifecycle to the component or components that implement it (eight with open-source-only coverage, three with mixed open-source/custom coverage and two with custom-only coverage), document the criteria (licence/sovereignty, fit-for-purpose, technology maturity) used to select each open-source component, and report operational telemetry captured from the live deployment, including per-container memory footprint, disk space, CPU usage, initial storage provisioning and the managed reverse-proxy and certificate-binding mechanism that keeps the independently built applications under one coherent identity and routing layer. The evidence shows that a lifecycle that would conventionally require several disjoint commercial services can be covered end-to-end on commodity single-host hardware (15 GiB RAM, 4 vCPU, 200 GB disk), at a measured aggregate memory utilisation of approximately 90% with zero configured swap, 16% CPU utilisation and 90 GB of storage in use, and that the custom components close specific gaps that the open-source components do not address on their own. We claim that the research gap this platform covers is not a missing set of tools but a missing reference architecture and evaluation for a small-footprint, interoperable, sovereignty-preserving, resilient data-science platform assembled from heterogeneous open-source and custom parts, and we propose a measurement matrix - covering completeness, interoperability, resource efficiency, security-control enforcement and reproducibility of deployment - for comparative evaluation across different installations. This is a single-instance, descriptive case report rather than a controlled experiment. The contribution is an architecture, an empirical operating profile and a replication mechanism, not a claim of superiority over any specific alternative.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-06
DOI
https://doi.org/10.5281/zenodo.23993134
Primary Topic
Scientific Computing and Data Management
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

IntelStack: An Open-Source, Sovereign and Resilient Data-Science Platform for Resource-Constrained Research Environments - Preprint

Chrysoula Samara, Spyros Ktenas
1 links
Zenodo (CERN European Organization for Nuclear Research)
Scientific Computing and Data Management
preprint

IntelStack: An Open-Source, Sovereign and Resilient Data-Science Platform for Resource-Constrained Research Environments - Preprint

Chrysoula Samara, Spyros Ktenas
preprint en

Abstract

Small and medium-sized organisations that need advanced data-science capability rarely have the budget, staff or legal capacity to adopt large commercial cloud platforms in a sovereign way. At the same time, single-purpose open-source tools deployed in isolation do not by themselves constitute a usable, governed data-science environment. This paper presents IntelStack, a sovereign and resilient platform that integrates multiple concurrently running applications on a single virtual machine with internal integration and unified governance: fourteen mature, carefully selected open-source projects (Apache Airflow, LinkedIn DataHub, DHIS2, Project Jupyter's JupyterHub, Keycloak, MinIO, NGINX, ODK Central, Ollama, Open WebUI, pgAdmin4, PostgreSQL, SNOMED International's Snowstorm, and Apache Superset) and five custom-built components (GlobalSecurity, GlobalStatus, IntelWeb, RWKV and SnowstormExtensionsAndAI) that supply identity-aware entry-point security, unified observability, Open-Source Intelligence (OSINT) capability, AI-assisted information analysis, public-facing dissemination, sovereign large-language-model usage and AI-assisted terminology extensions. The platform was developed and first operationally tested in the context of the research project "Interoperability and utilisation of epidemiological surveillance data with artificial intelligence", which required the rapid development of a low-cost, standalone/sovereign and resilient data-science environment for public-health data experimentation. The architecture, however, is domain-agnostic and is reported here as a general-purpose sovereign data-science platform rather than as a public-health-specific tool. We describe the platform's layered architecture (acquisition, orchestration, storage, cataloguing, semantic enrichment, prediction/model execution, analytics, dissemination, identity, entry-point security and observability), map each of the thirteen stages of a generic data-science lifecycle to the component or components that implement it (eight with open-source-only coverage, three with mixed open-source/custom coverage and two with custom-only coverage), document the criteria (licence/sovereignty, fit-for-purpose, technology maturity) used to select each open-source component, and report operational telemetry captured from the live deployment, including per-container memory footprint, disk space, CPU usage, initial storage provisioning and the managed reverse-proxy and certificate-binding mechanism that keeps the independently built applications under one coherent identity and routing layer. The evidence shows that a lifecycle that would conventionally require several disjoint commercial services can be covered end-to-end on commodity single-host hardware (15 GiB RAM, 4 vCPU, 200 GB disk), at a measured aggregate memory utilisation of approximately 90% with zero configured swap, 16% CPU utilisation and 90 GB of storage in use, and that the custom components close specific gaps that the open-source components do not address on their own. We claim that the research gap this platform covers is not a missing set of tools but a missing reference architecture and evaluation for a small-footprint, interoperable, sovereignty-preserving, resilient data-science platform assembled from heterogeneous open-source and custom parts, and we propose a measurement matrix - covering completeness, interoperability, resource efficiency, security-control enforcement and reproducibility of deployment - for comparative evaluation across different installations. This is a single-instance, descriptive case report rather than a controlled experiment. The contribution is an architecture, an empirical operating profile and a replication mechanism, not a claim of superiority over any specific alternative.

Zenodo (CERN European Organization for Nuclear Research)
University of Thessaly (GR), Ionian University (GR)
Industry, innovation and infrastructure
Scientific Computing and Data Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon