Observability: A Practical Primer for RSEs

When you distribute and run your software, do you have a good idea of how it's functioning "in the wild"? Do your users have to manually report back to you when they find a bug - and do you have access to the software state when it occurred? How do you know if your application has enough resources when running on a cluster or the cloud, and at what rate it's processing datasets/jobs?These questions are some of the motivating use cases underpinning observability, which can broadly be defined as using event-driven data (telemetry) to gain insights into the functioning of both software and infrastructure. Observability is useful for troubleshooting issues on production systems, monitoring performance and security, and for tracking key metrics and events happening within a system.In this introductory walkthrough, I will introduce observability in the context most often used by RSEs - running software in "batch" mode (e.g. running on a cluster) or "server" mode (long-running applications on cloud or workstations). The three main telemetry types will be introduced - metrics, logs, traces - and I will demonstrate how to capture metrics from Python applications in "batch" mode using Microbench (pypi.org/project/microbench) and in "server" mode using OpenTelemetry (opentelemetry.io) and Prometheus (prometheus.io). I will share resources for capturing logs and traces (for later study, in the interests of time). The emphasis will be on simple, practical examples that RSEs can adopt into their own software with minimal fuss.A recording of this session is available on YouTube: https://youtu.be/-FtIMrmHApg