Systems Engineering of Adaptive AI Inference Orchestration Across Heterogeneous Accelerators

ABSTRACT Heterogeneous AI accelerators solve the fundamental problem of limited compute capacity and rigid pricing by enabling access to diverse computational resource pools with varying cost‐performance characteristics. This mirrors federated cloud computing paradigms where resource pooling across providers optimizes utilization and cost. We present an SE framework for adaptive orchestration using hierarchical decomposition, standardized interfaces, and model‐based engineering. Our approach integrates INCOSE lifecycle processes with evolutionary architecture patterns to manage technical complexity while enabling stakeholder value. Empirical validation achieves 70% cost reduction and 99.95% availability, demonstrating how SE principles enable sustainable heterogeneous AI systems.