Data Lakehouse: Building the Foundation for AI/ML Ready Scientific Data

Scientific research increasingly depends on combining data from multiple sources, such as molecular structure data from one repository and energy calculations from another to train AI/ML models and perform cross-dataset analysis. To that end, a researcher should locate where each dataset is stored and in what format, download the data in the specific format, write custom code to parse and load each format using different libraries, clean and normalise the data, and prepare special code for schema alignment before the analysis even starts. Such data engineering steps should be performed for each experiment, which will be an additional burden on researchers and scientists. PSDI Data Lakehouse was designed and implemented to address this challenge by providing a queryable platform with well-aligned and interoperable scientific datasets, including OMol25, OMat24, Materials Project, and Materials Project Contribs. PSDI Data Lakehouse is based on an architecture with the use of Apache Iceberg as an open table format, Lakekeeper as a REST catalogue, Trino as a distributed SQL query engine, and Ceph S3 as the underlying object storage, organised around a Bronze-Silver-Gold medallion pattern. Data initially distributed in different formats, including ASE LMDB databases and JSON, are converted, validated and registered as Iceberg tables which can be queried using SQL and Python without the need for a researcher to develop any additional parsing or schema-alignment code. By leveraging open-source technologies, the PSDI Data Lakehouse breaks down barriers to accessing diverse data sources, fosters interoperability, and empowers AI/ML-driven research.