Design Principles and Guidelines for LLM Observability: Insights from Developers

The growing number of large model development tools has increased the demand for transparency and accountability. We present four developer-centric design principles for large language model (LLM) observability across the entire LLMOps lifecycle. The research focuses on improving the developer experience by integrating design principles that address the unique needs of developers at various stages of LLM development. A mixed-methods methodology was implemented consisting of three focus groups (n=10 per group) with developers across different proficiency levels, complemented by a small-scale designer rating survey. The study highlights key design principles—Design for Awareness, Monitoring, Intervention, and Operability—and their impact on developer experience factors like mental models, efficiency, visualization, and understandability. The findings reveal significant correlations between these principles and developer needs, with a particular trade-off between efficiency and mental model clarity. The study provides actionable insights for designing effective observability tools and emphasizes the importance of balancing developer control, transparency, and ease of use.