Performance evaluation for detection, tracking, and identification (DTI) systems

This paper reports on a novel verification and performance evaluation framework specifically designed and developed to facilitate a standardized comparative performance evaluation for commercial detection, tracking and identification (DTI) solutions to counter Unmanned Aerial System (UAS) threats. The test methodology is designed to compare commercial systems in a fair and reproducible manner based on end-user defined criteria. <br/> <br/> DTI systems are increasingly relevant for e.g., perimeter protection of military facilities, critical infrastructures and public events and the expected end-users are law enforcement agencies, the military, civil defense agencies and private entities. However, such systems are commonly hard to benchmark in a fair and comparable manner and performance claims of these systems are currently not supported by evidence. In addition, no standardized test methodologies are currently available making it near impossible to compare competing DTI systems. <br/> <br/> In Courageous we developed an objective driven test methodology for use by the civilian sector. Courageous leads to a comparative performance evaluation system for commercial DTI solutions for Counter-UAS systems (C-UASs) using operationally relevant end-user scenarios and a generic DTI system lay-out. The work takes into account contextual information as well as end-user input, albeit focusing primarily on civilian use cases so far. We outline the process taken as well as the resulting system and discuss how the systems should be evaluated and validated iteratively over time. We furthermore elicit end-user input from the defense domain and argue that the scope of Courageous should be broadened to include military challenges, aspects and concerns. <br/> <br/> The work with regard to homeland security use-cases, presented here, has firstly been verified in a simulation environment where a number relevant scenarios were used and the output of the simulation injected into the testing system. Validation of the work in a relevant environment has been done in three operational trials. <br/> <br/> The results from the operational trials held for homeland security scenarios show that the method allows for performance evaluation at component level (i.e., detection, tracking or identification component) and at system level (combinations of these components and integrated DTI system of system solutions).