Investigating algorithm review boards for organizational responsible artificial intelligence governance

Organizations are increasingly developing and utilizing artificial intelligence (AI) tools. Responsible AI (RAI) governance approaches including algorithm review board (ARBs) have emerged as important mechanisms to address potential AI risks and harms. We interviewed 17 technical contributors at a variety of organizations about their experiences with internal RAI governance. We summarized the first detailed findings on ARBs in practice, including their membership, scope, and measures of success. We confirmed known ARBs in finance sectors and revealed extensive use of ARBs in health sectors. Our findings suggest that Institutional Review Boards alone are insufficient for algorithm governance and that ARBs are used in tandem with other RAI approaches. Integration with existing processes and leadership buy-in were considered critical to the success of internal RAI governance. Financial tensions between profit and the cost of RAI were a major concern for many participants. Future work should explore measurements of success for ARBs and expand analysis of ARBs to other industries leading in AI implementation.