Measuring stakeholder influence: a comparison of automated and manual approaches

Democratic governments routinely consult with those in society who would be affected by policy changes before they go into effect. The extent to which stakeholders’ preferences influence government policies, however, can vary considerably. Measuring interest organisations’ influence in consultations is therefore an important research endeavour. We generate a dataset encompassing more than 1,000 submissions from stakeholders in response to consultations carried out with respect to Danish bills in the parliamentary year 2021/22. By comparing these submissions against official government statements, we measure influence in three distinct but comparable ways: 1) manual coding of influence, 2) a text reuse approach, and 3) AI chatbot-assisted coding of influence. In applying each of these to the same data, we draw three key conclusions. First, each method has advantages (and disadvantages). Second, text reuse methods can find compelling evidence of stakeholder influence, even if it does not identify all influence. Third, and most innovatively, we demonstrate that AI-assisted influence measurement shows considerable promise by reducing the time and effort required to identify influence while achieving similar, and possibly more objective, results relative to manual coding. This study helps move the discipline toward better understanding of outside influence in government decision-making.