Evaluating the Reliability of Pragmatic Annotation Guidelines for Ritualized Expressions in Spanish through Corpus and Ontology-Based Analysis

This study addresses the theoretical and methodological challenges of identifying and classifying ritualized expressions in Spanish, specifically greetings, farewells, vocatives, exclamatives, and politeness formulas, by developing and applying a dedicated annotation framework. Its principal aim is to evaluate the reliability of a set of annotation guidelines based on an Ontology of Ritualized Expression (REO), applied to a corpus derived from Spanish-language multimedia content. The study adopts an interdisciplinary approach, drawing on phraseology, pragmatics, discourse analysis, and applied linguistics, with a particular focus on Spanish as a Foreign Language (ELE). Ritualized expressions, or pragmatemes, are conventionalized linguistic units whose meaning largely depends on the communicative situation (SIT) in which they occur. Although such expressions have received considerable attention in lexicographic and theoretical literature, their pedagogical integration in ELE textbooks remains inconsistent. In many CEFR-aligned ELE manuals, these expressions are inadequately described or contextualized. Sociopragmatic variables—such as diatopic variation, deixis, register, and situational appropriateness—are often underrepresented or insufficiently addressed in instructional content. This research builds on recent developments in phraseology and pragmatics, including studies on affective pragmatemes, typology of diatopic variation, and foundational models from the explanatory-combinatorial framework. Previous corpus-based investigations, particularly those involving the analysis of audiovisual dialogues and ELE instructional texts, serve as important references. The study comprises three integrated phases: 1. Corpus selection and annotation: A working corpus was compiled using three Spanish-language films in their original version (VOE), using both subtitles and their corresponding audiovisual scenes. These materials provided naturally occurring instances of ritualized expressions in discourse. Selected segments were chosen for their rich pragmatic content. 2. Ontology-based annotation: The Ritualized Expressions Ontology classifies ritualized expressions into five main categories: greetings and farewells, vocatives, exclamatives, politeness formulas, and profanities. Each category is further subdivided according to situational, formal, and functional parameters. Expressions were annotated using a labeling system that facilitated nuanced classification by contextual appropriateness, degree of ritualization, and interactional function. 3. Validation and reliability testing: A pilot annotation study designed to assess the robustness of the guidelines and inter-annotator consistency was carried out. Six annotators participated: Five human annotators, representing a diversity of linguistic backgrounds; three Spanish native speakers, serving as a baseline, one Portuguese and one Spanish speaker, with specialized lexicographic knowledge, and the generative AI model ChatGPT 4.5. Human annotators received printed excerpts from the three selected film subtitles and were given access to the corresponding audiovisual scenes. They manually annotated instances of ritualized expressions according to the REO guidelines. The subtitles and the annotation guidelines were also submitted to a customized version of ChatGPT 4.5 (OpenAI, 2025) for evaluation and comparison. Upon completion, an extensive cross-comparison was carried out, measuring the reliability index of annotations through pairwise comparisons among all annotators (human-human, human-AI, and AI-human). These comparisons constitute the primary dataset for assessing guideline’s reliability. The findings underscore the importance of rigorously defining and operationalizing the communicative situation (SIT), particularly in pedagogical materials. The REO framework offers a structured yet adaptable framework, able to account for variation across multiple dimensions: medium (oral vs. written), dialect, formality, emotional register, and frequency. Lemmatization techniques were also applied to standardize surface forms and enhance annotation consistency. From a didactic standpoint, this research highlights the urgent need for ELE textbooks to adopt more context-sensitive approaches to teaching ritualized expressions. Incorporating pragmatic features—such as emotional register, speaker hierarchy, social distance, and appropriate first responses—can significantly improve learners’ interpretative and expressive skills. Moreover, this research contributes to computational pragmatics by evaluating how LLMs can support annotation tasks while also revealing their limitations in handling situational nuance. Comparative reliability data involving human annotators and AI models highlight the promise and the current shortcomings of such tools in linguistic research. In conclusion, this study establishes a methodological foundation for evaluating annotation guidelines in discourse pragmatics and proposes a scalable, ontology-based framework for building annotated corpora centered on ritualized expressions. It opens new avenues for cross-linguistic and cross-register pragmatic studies and suggests practical applications in ELE curriculum design, lexicography, and AI-driven language technologies.