Character recognition in Byzantine seals with deep neural networks

Seals are small coin-shaped artifacts, mostly made of lead, held with strings to seal letters. This work presents the first attempt towards automatic reading of inscribed text on Byzantine seal images. Byzantine seals are generally decorated with iconography on the obverse side and Greek text on the reverse side. Text may include the sender’s name, position in the Byzantine aristocracy, and elements of prayers. Both text and iconography are precious literary sources that wait to be exploited electronically, so the development of computerized systems for interpreting seals images is of paramount importance. This work’s contribution is hence a deep, two-stages, character reading pipeline for transcribing Byzantine seal images. A first deep convolutional neural network (CNN) detects characters in the seal (character localization). A second convolutional network reads the localized characters (character classification). Finally, a diplomatic transcription of the seal is provided by post-processing the two network outputs. We provide an experimental evaluation of each CNN in isolation and both CNNs in combination. All performances are evaluated by cross-validation. Character localization achieves a mean average precision (mAP) greater than 0.9 at the intersection of union threshold of 0.5. Classification of characters achieves an accuracy greater than 0.92. Such performance compares favorably to similar tasks such as the recognition of inscribed characters on ancient coins. At transcription level, we provide novel performance results in terms of Character Error Rate. This is novel for seal images and differs from results on isolated character recognition. • We present the first system dedicated to the recognition of inscribed characters in Byzantine seal images. • A two-stages deep architecture is proposed for localizing and classifying characters. • A character-level diplomatic transcription is then obtained (by reordering characters into text lines) as the first step towards language processing to extract words, named entities and recover the underlying text.