A Classification Benchmark Based on the Literary Theme Ontology

We introduce two new datasets of TV episode summaries (n = 644) and subtitles (n = 956) in English, human-annotated with one or multiple themes. The datasets are derived from the Literary Theme Ontology, an ontology that uses well-defined definitions of themes to identify themes in stories. This multi-label classification task is then tested on bag-of-words classification models, as well as small open-weight LLMs. Since themes in TV series episodes do not have to be explicitly mentioned in a summary or in the subtitles, and the themes themselves can be rather abstract, the theme classification is a hard task. SVM classifiers are most successful at predicting the themes in TV episode summaries and subtitles (F1 = 0.50 and 0.44). The results also show that the length of the input text strongly influences the ability of the LLMs to follow the instructions given in the prompt, and answer in the provided output format.