White Elephants and Cash Cows: Economically Wrangling the Zoo of AI Models

The capabilities of artificial intelligence are rapidly expanding, but deploying AI systems in practice still poses significant challenges. Specifically, practitioners find limited guidance on selecting the most suitable AI model for a concrete use case, balancing the economics of an AI deployment, and managing the risk of AI errors. These challenges call for a unified framework addressing pain points in a conceptually clear and statistically sound manner. In this thesis, we present several components of such a framework: 1) uncertainty-aware system optimization, 2) economic evaluation, 3) error reduction with human-in-the-loop, and 4) a proof-of-concept system for synthetic data generation. Our work presents novel technical and conceptual approaches for orchestrating natural language-based systems, advancing the economical and reliable deployment of artificial intelligence.