Investigating Performance Overhead of Distributed Tracing in Microservices and Serverless Systems

Distributed tracing is crucial to achieve observability in modern distributed systems. However, its adoption introduces performance trade-offs, impacting throughput and latency. This paper investigates the overhead of distributed tracing in microservices and serverless applications. We provide an analysis of the popular OpenTelemetry and Elastic APM distributed tracing frameworks, evaluating their performance impact on microservices and serverless workloads. We highlight and categorize the primary sources of overhead and measure their contribution to performance degradation. The results reveal significant throughput reductions (19-80%) and latency increases (up to 175%) depending on application configurations and execution environments. Our findings reveal that serializing trace data for export is the largest cause of overhead.