Enhancing Distributed System Reliability through Request-Level Fault Injection and Fine-Grained Tracing

Ensuring reliability in microservice architectures requires sophisticated fault injection methods to test system resilience. Traditional approaches to fault injection either rely on random failure generation, which can overlook critical faults, or involve complex modifications to low-level interfaces, risking further instability. This paper introduces Request-Level Fault Injection (RLFI), a novel approach leveraging the baggage annotation capability of the Opentracing framework to propagate failure flags at the request level without altering lowlevel protocols. By integrating RLFI with tracing, we gain comprehensive insights into system behavior under fault conditions, allowing developers to diagnose and address failure points effectively. RLFI supports fine-grained control over fault scenarios by combining decorator-based injection with Lineage Driven Fault Injection (LDFI), offering a robust framework for debugging distributed systems. Our empirical results show that RLFI enhances fault detection rates and reduces system impact compared to traditional methods, enabling efficient and precise testing in complex architectures. This work represents a significant advance in fault tolerance for microservices, presenting a practical, low-overhead method to ensure service reliability and availability.