Holistic Integration of Performance and Programmatic Metrics in Analysis of Alternatives

Initiating and completing an objective and robust Analysis of Alternatives (AoAs) when developing a capability, system, or product is influenced by the level of importance of each technical metric (Key Performance Parameter) or desirable attribute. This can be driven by mission, system or product need, stakeholder surveys or system engineers' ad hoc discussions. An AoA as defined by DoD policy is an analytical comparison of the (mission) operational effectiveness, suitability, risk, and life cycle cost of alternatives under consideration to satisfy validated capability needs and affordability. When balancing technical and programmatic performance, then combining them into a single weighted score can be difficult. Moreover, results can differ if weighting allocations change based on stakeholder priorities or influences that change alternative choices. An approach to developing an objective metric for AoAs, are often expressed in terms of technical metrics using the Joint Capabilities Integration Development System (JCIDS) such as measures of effectiveness (MOEs), measures of suitability (MOSs), or measures of performance (MOPs) among others. They contain multiple technical parameters (e.g., survivability, reliability, availability, range, field of view, etc.) that can be integrated into one overall measure to compare solutions. Key programmatic metrics considered though not often explicit in AoA solutions are schedule, technical maturity, quality, risk, security, and logistics, etc. To support a complete unbiased solution, we provide a framework using the analytical hierarchy process to organize and objectively integrate attributes. Technical and programmatic metrics are combined, into a single measure. The results can then be expressed as a comprehensive group of solutions plotting them on the “Y” axis, (ordinate) against cost on the “X” axis, (abscissa) within an affordability range. It is like cost as an independent variable (CAIV) where finding the “sweet spot” balance between performance and cost. Then we expand the trade space into a three-dimensional solution to highlight some of the more important requirements such as programmatic performance, risk or logistics to provide additional sensitivity from an integrated program management perspective. These visualizations can reveal other drivers not always shown in a holistic solution. We draw from historical system engineering research by Saaty, Keeney and others on technical metric integration, then expand on their work to provide a total integrated technical and programmatic solution that can be applied to a variety of AoAs. Developing representative life cycle cost estimates (LCCE) for each design will accompany each alternative. The LCCEs will be structured using a work breakdown structure integrated with the key metrics, then risk adjusted to provide probabilistic solutions for added fidelity. The results of this methodology will clearly highlight drivers not always shown in a AoA that can influence solution choice or cost and schedule uncertainty when making materiel decisions. Future work will investigate how to consider delaying a final solution “lock in” too early to efficiently integrate technical and programmatic knowledge, address attribute weighting and scalability, investigate the influences of machine learning and validated large language models to support high fidelity AoA studies.