EDGAR-CRAWLER: From Raw Web Documents to Structured Financial NLP Datasets

Acquiring data for resource-limited domains like financial nlp presents numerous challenges.One rich web resource of textual financial data is edgar, the Electronic Data Gathering, Analysis, and Retrieval system, where U.S. legislation requires publicly traded companies to upload economic reports for investor transparency.However, automating the process of downloading edgar filings is not enough.These long-form documents, often exceeding 100 pages each, include noisy text, images, and tables, requiring significant manual preprocessing.Moreover, researchers often require only specific sections of these lengthy documents to solve specific nlp tasks, further complicating the use of these filings.To address these challenges, we present edgar-crawler, the first Open-Source Software (oss) for automatically downloading and preprocessing edgar filings.Equipped with a variety of document mining options, edgar-crawler can convert lengthy raw financial documents into clean and section-specific json files, which can be integrated seamlessly into nlp workflows.Supporting multiple filing types and offering extensive filtering options, edgar-crawler facilitates the creation of structured large-scale datasets for financial research.Serving as a free alternative to premium web service data providers, edgar-crawler has been widely adopted by both practitioners and academic researchers.Our software is available at https://github.com/lefterisloukas/edgar-crawler.