TRUJAMAn: Word-Level Recognition of Judeo-Arabic and Hebrew in Medieval Texts

Once transcriptions of the Genizah letters have been obtained and dated, the next step is to create a method for reliably identifying the languages used in each letter. Given that there are more than a thousand letters in the dataset, tagging the language of each letter manually would require a long time or a large team of experts – possibly both. Logistic regression has already been used as a method for language identification. A model trained in this way identifies the most likely or unlikely characters or combinations of characters for two languages, and turns them into a probability. This method has the advantage that, after training, weights can be reviewed, which allows us to understand better the parameters on which the model's decisions are based. For this kind of approach, large quantities of data are required. The kind of texts needed are medieval works that include sections written in Hebrew and Judeo-Arabic, and labelled as such. Surprisingly enough, such a specific dataset exists. The whole dataset comprises 110 works written by 36 authors. 70 works by 11 authors were used for training, and 40 works were set aside for evaluation. Of these, 31 works are by authors the model never saw during training. The upper dot Judeo-Arabic diacritic was removed before training. This means that language classification will work regardless of diacritic presence or consistency, which tends to vary by author and also depending on transcription criteria. This model provides a baseline for the problem of language identification in mixed texts which contain Hebrew and Judeo-Arabic. Every metric scores higher than 90%, and its accuracy is at 96.4%. Another possible test is to compare the results for language labelling at the document level. If this evaluation is carried out only on transcriptions from the Princeton Geniza Project (PGP), which are human-made, the percentage of letters correctly identified stays in the higher 90s. Once the creation date of several documents and their language share – what percentage of their words are Judeo-Arabic, and what percentage Hebrew – are known, these two values can be used as coordinates on a graph. This opens new research possibilities, as it allows us to identify trends in the Genizah and the documents that are most representative of them – or the ones that escape them.