Finding Faults Faster: Improving Error Tracing in Reactive Monitored Systems with Distributed Tracing Tools

In modern systems composed of many interconnected components, detecting and diagnosing errors has become increasingly complex. For proactively monitored systems, where data is continuously collected and analyzed in real time, there exists well-established observability solutions. These systems can track performance metrics, generate alerts when anomalies are detected, and even visualize request fows across services correlated with logs as they happen, enabling rapid detection and response to issues.However, in environments that rely on reactive monitoring, this level of real-time visibility is not possible. Instead of continuously collecting observability data and provide real-time monitoring, these systems depend primarily on analysing logs to understand what went wrong, and only after the problem has already occurred.An important part of debugging relies on the logging itself. A pre-study was conducted to gather information about existing logging practices and challenges. Insights gained from pre-study and user study were compiled into a set of logging guidelines for Saab aimed to improve log quality and consistency across systems.The thesis also explores whether distributed tracing tools are applicable in reactive monitored systems and whether they can help improve error identifcation and debugging. A prototype system meant to serve as a representation of a system at Saab was implemented. The implemented system was instrumented with distributed tracing techniques and tools, and a user study was conducted to evaluate the aid to fnd the root cause of errors in a system with the use of these tools.Participants of the user study were asked to locate bugs in diferent scenarios, with and without the aid of visualization tools. The results indicated that the use of distributed tracing tools could help provide a valuable overview of the system status and potentially help to fnd root causes of errors more efciently. However, the idea and attempts to construct visualized traces solely from enriched logs were found to be impractical due to the complexity of the data required and the process it would require to ingest the data into the visualization tools.