Evolution of Extract-Transform-Load (ETL) processes towards data product pipelines

The rise of data as a first-class asset has led to creating infrastructures and tools designed to enhance organizations’ abilities to monetize them internally. One of the most powerful tools have been ETLs, which govern the internal data operations, assisting in companies in their quest to becoming data-driven. Lately, these horizons have expanded with the apparition of new ecosystems for data exchange, such as Data Spaces or initiatives like Gaia-X or SIMPL, allowing companies to monetize data externally, e.g. sharing or selling them. However, traditional ETLs fall short to serve this purpose. In this article, we try to offer a technological comparison of how current ETL tools are prepared to address the new concept of data pipeline aimed at achieving a data product. Furthermore, this comparison is proposed within the framework of a project like DATAMITE, which allows it to be provided with real scenarios and use cases, in which its benefits and applicability can be accurately appreciated.