Don't Confuse Engagement With Leadership: Reconstructing Decision-Making Dynamics During Public Health Crises by Applying Data Science to Public Records (Supplementary Material)

Overview Supplementary material describing the data science pipeline used in "Don't Confuse Engagement With Leadership: Reconstructing Decision-Making Dynamics During Public Health Crises." It offers a practical framework for researchers and data scientists facing the challenge of analyzing massive, unstructured public records datasets. Intended Audiences Data Scientists interested in applying Computer Vision and NLP to real-world government datasets. Public Policy researchers seeking efficient ways to explore large repositories of digital correspondence. Topics Automate Document Classification: Learn how to train Vision Transformer (ViT) models to automatically filter out clutter and sort documents into classes like emails, spreadsheets, and handwritten notes. Reconstruct "Lost" Metadata: See how to build a Named Entity Recognition (NER) workflow to extract senders, recipients, and dates from raw image scans, reconstructing decision-making timelines. Validation: Learn about techniques for validating the performance of document classification and NER on large datasets. Move Beyond Keyword Search: Discover how to implement semantic search using vector embeddings, allowing you to query the context of documents (e.g., "local engagement") rather than just exact text matches. Deployment: Includes details on deploying these models into a user-friendly web interface using Streamlit and PostgreSQL, making complex data accessible to non-technical experts.