Mentorship and Leadership in Data Engineering Teams: Fostering Growth and Knowledge Sharing

The fast-paced field of data engineering demands adaptive leadership and strong mentorship to enable growth and knowledge sharing. As organizations move toward data-driven decision-making, the leadership function in data engineering teams has gained more importance. Transformational and servant leadership styles, for example, are demonstrated to impact team innovation, motivation, and autonomy. Formal and informal mentorship are crucial for the professional growth of data engineers, allowing the exchange of technical and soft skills, as well as improving team performance and cohesiveness. While there have been major leaps in the knowledge of leadership and mentorship, research on how these elements uniquely interact in data engineering teams is limited. This review aggregates evidence from 2015 to 2024 and highlights the most important aspects of leadership and mentorship that foster growth and collaboration among data engineering teams. It underscores the significance of emotional intelligence, effective communication, and peer mentorship in building a culture of knowledge sharing. In addition, as data engineering teams continue to embrace agile methodologies and work in distributed teams, leaders need to shift their strategies to keep the team cohesive and learning continuously. The research gap exists in the absence of holistic studies that investigate leadership and mentorship within the context of emerging technologies like artificial intelligence and machine learning. Moreover, the effect of diversity and inclusion practices on team performance and knowledge sharing has not been adequately examined. This review calls for additional research to fill in these gaps and offer actionable recommendations for organizations that are attempting to optimize the performance of their data engineering teams.