Evaluating Embedding Models for Retrieving ESG Information from Annual Business Reports

Interest in Environmental Social Governance (ESG) and Sustainable Development Goals (SDG) has become paramount. Companies embed their ESG achievements, plans, and commitments into their annual reports in a non-standardized textual format. We propose a retrieval system for increasing transparency and comparability in ESG reporting, enabling various stakeholders to compare sustainability statements across annual reports. We employed qualitative and quantitative evaluation methods to select the most suitable embedding model for the proposed ESG retrieval application. Our results show that the domain-relatedness of models has no positive impact on the retrieval performance in the ESG domain. However, domain-related evaluation datasets support a more relevant approach for model evaluation in the domain of interest. We have developed a sustainable and efficient benchmark for ESG-related retrieval systems that is faster and more resource-efficient than generic benchmarks, which typically include many datasets that are not relevant to the domain at hand.