Measuring Firm Characteristics with Natural Language Processing (NLP) Models: An error-mitigating framework with tutorial guidance

This paper develops an error-mitigating framework, including tutorial guidance, with respect to using natural language processing (NLP) and language models to measure firm characteristics from web-scraped data, such as company websites. Over the past decade, the digital presence of firms has expanded, offering researchers new opportunities to capture organizational attributes that were previously difficult to measure at a large scale. With our framework, we seek to offer three key contributions. First, we develop a taxonomy of challenges—categorized into representation errors, measurement errors, and computational errors—associated with applying NLP to firm-level web data. Second, we formulate recommendations for preventing, managing, and reporting these challenges. Drawing from the total error framework, we address potential biases, validity issues, and computational difficulties throughout the process, from data collection to data analysis. Third, using the Dutch FIRMBACKBONE initiative as an example, we illustrate the practical applications and obstacles in building reliable digital infrastructures for firm data research. The paper concludes by highlighting the importance of investment in data infrastructures to create secure, and pseudonymized datasets for academic research.