Shadow testing for on-prem Hadoop System & AWS EMR

As enterprises transition from on-prem Hadoop to cloud-native platforms like AWS EMR, ensuring performance parity and operational efficiency is critical.This paper introduces a custom Shadow Testing Framework designed to compare production workloads across both environments.By replaying real queries from on-prem systems in EMR, the framework captures execution metrics, identifies configuration gaps, and supports parallel testing across multiple clusters.Key architectural shifts-such as moving to Tez, S3 storage, and enhanced SQL authorization-are addressed.Our results show that, with tuning, EMR can meet or exceed on-prem performance while delivering scalability, reliability, and cost benefits.This approach enabled a productiongrade EMR deployment supporting thousands of daily queries.