Building a historical disease database with newspapers: Measuring the prevalence of infectious diseases across time and space in the Netherlands, 1830-1940

The nineteenth and early twentieth century Netherlands had a very high burden of infectious diseases. Outbreaks of diseases such as tuberculosis and smallpox were common. To study health and living standards in this period, capturing the disease burden is critical.While today we are able to measure the disease burden with infection rates, this is not possible for the past. In the Netherlands, there are no available measures of the disease burden prior to the mid-twentieth century, aside from mortality rates.Mortality rates are also not perfect proxies of disease: they fail to capture diseases that negatively impacted an individual throughout their life, but did not kill them. This prevents researchers from, for instance, studying how exposure to infectious diseases in early life might impact someone’s health in later life. An indicator of the infectious disease burden that is not based on mortality information would enable researchers to do so.In the nineteenth and twentieth centuries, Dutch newspapers reported on the number of new cases and deaths from specific infectious diseases. A large share of these newspapers are digitized and publicly available via the Royal Library’s Delpher.nl. This project aims to convert these newspaper mentions into a quantitative indicator of the disease burden.To do so, we leverage natural language processing methods to: process newspaper mentions of a disease, connect them to a geographical region, map them onto numeric scales, and assess their performance as a proxy of the disease burden in a given municipality and year. The result will be an open-source disease database and historical map. Further, we show how this disease database can be used by researchers, and how to account for measurement uncertainties in the disease database.