An approach to establishing evaluation criteria for unmanned surface vessel manoeuvres during COLREGs encounters

Abstract The question arises when developing and testing Unmanned Surface Vessel (USV) Manoeuvring Autonomy (MA): ‘is the performance we are seeing in our current on-water tests better than that of the last autonomy software version we deployed?’ An approach to answer this question is inspired by educators’ rubrics, in which a teacher grades a student’s work to objective criteria and then sums the individual criteria to determine the student’s overall grade. Here, individual metrics are used to evaluate a USV manoeuvring within range of another vessel. A weighted average is then applied to determine the overall score. With that objective performance value now obtained, similar manoeuvring tests can be compared between autonomy software versions to determine if the autonomy under development is progressively improving. This paper does not determine the threshold score needed to establish that a USV is safe to operate; thresholding of sufficient performance is recommended for future study.