Guidelines for Data Engineering Documentation in a DevDocOps Approach

The software development process has been studied since the beginning of technological evolution. Development practices have evolved, requiring processes capable of supporting intensive work, paving the way for agile methodologies. With the growing need for continuous integration (CI) and continuous deployment/delivery (CD), new data architectures have emerged, allowing for scalable, maintainable, and reusable environments, collectively known as DevOps (Development + Operations). In this context, the DevDocOps approach integrates continuous documentation into the software development lifecycle. However, little has been published regarding the benefits of this approach. To address this, an empirical study was conducted, applying findings from the literature to a real development environment by integrating continuous documentation into the data engineering development lifecycle. Based on feedback from developers and technical lead, the results highlight the importance of technical documentation in an agile development environment and demonstrate how automating this process can improve the quality and efficiency of software deliveries.