Elusive reliability: Techno-legal considerations on the challenges of maintaining and reviewing performance of AI in military applications

This article examines the reliability of military artificial intelligence (AI) systems and the implications thereof for compliance with international humanitarian law (IHL) amidst increasingly non-linear and experimental nature of AI retraining and fine-tuning. It demonstrates that various machine-learning techniques present a significant risk of performance degradation when used in warfare. Offline learning systems, while static and manually updated, suffer from concept drift as battlefield conditions evolve, degrading performance without proactive human intervention. Online learning, though adaptive in dynamic environments, is prone to catastrophic forgetting and adversarial manipulation, undermining predictability. Central to the analysis are user-generated feedback loops in offline systems, intended to refine performance through iterative updates. Drawing on interviews with military engineers and fieldwork, the article illustrates how anticipated cyclical refinement of AI systems undermines their reliability and users' ability to assess compliance of their use with IHL. The cycles of optimization, instead, introduce unpredictability. The article’s significant contribution lies in bridging the technical realities of AI systems with IHL compliance frameworks. It critiques the assumption that reliability is a machine-driven process, instead highlighting the dependence of machine's "learning capabilities" on the Sisyphean labor of tinkerers who adapt and maintain the performance of AI models. By unpacking how offline learning updates and feedback loops destabilize legal reviews, the analysis calls for rethinking Article 36′s application to AI-enabled tools, emphasizing continuous monitoring and transparency in system modifications. This work provides a foundation for legal scholars to engage with the socio-technical complexities of military AI, ensuring accountability amid evolving battlefield technologies.