Developing a TEI-Encoded Digital Archive of 19th-Century Urdu Marsiya: Production and Challenges

The Text Encoding Initiative (TEI) was established in 1987 to provide guidelines for the creation and management in digital form of every type of data created and used by researchers in Humanities. Despite its popularity and usage over the past 25 years, TEI has rarely been applied to South Asian languages, particularly in Urdu, which contains a significant amount of literature in its original manuscript format. These manuscripts are circulated as scanned, un-editable, and static PDFs. Despite its rich literary tradition, Urdu is facing a decline in accessibility and readership, therefore, there is a pressing need for digital editions of Urdu literature. The problem with ancient and pre-modern Urdu literary texts is in their utilisation of distinctive calligraphic techniques, often lacking any references in their first prints. This results in a two-dimensional text, typically distributed in printed format or scanned PDFs. Due to a decrease in the availability of Urdu learning courses, authorities, and platforms, individuals who are more proficient in the sister-language, Hindi, face challenges in independently reading Urdu script without the aid of professional guidance. The lack of digitization of old Urdu literature, with its unique and unfamiliar calligraphic forms and absence of metadata, is a challenge for young Urdu language users to fully appreciate and understand the nuances within the texts. A digital archive of Urdu literature can significantly mitigate the issues by providing accessible, interactive, and enriched textual experiences. Text encoding with TEI standards can include comprehensive metadata, contextual annotations, and translations which help overcome the barriers posed by traditional calligraphic forms. This paper outlines the development of the first TEI-encoded digital archive of Urdu Marsiya (elegies), focusing on a significant Urdu corpus from 19th-century Lucknow. These elegies, commemorating the Battle of Karbala and the martyrdom of Prophet Muhammad's grandson, are pivotal to Urdu literary heritage. The approach utilizes an Optical Character Recognition (OCR) engine to transcribe these texts from their original Perso-Arabic script. This transcription process involves significant manual correction to accommodate the unique calligraphic styles of Urdu, which often omit essential diacritical marks and employ context-specific letterforms. Following transcription, the texts are encoded according to TEI standards, facilitating detailed linguistic and stylistic analysis. I further discuss the technical challenges encountered such as issues with OCR accuracy due to the script’s right-to-left orientation of the Perso-Arabic script, the inadequate word separation in Urdu and complex calligraphy. I outline the technical solution implemented, which includes a programmatic way of (i) automating text extraction from PDFs using OCR technology (ii) extracting relevant entities such as people, places, events, concepts used in the text using Named Entities Recognition techniques in Natural Language Processing (NLP) and finally, (iii) transforming the extracted text into a TEI encoding. The broader implications of this work for the field of Digital Humanities in South Asia and the preservation of regional languages are also considered, highlighting the project's contribution to global scholarship and the promotion of Urdu literary studies.