Using human-Al collaboration to explore meanings of semiotic resources in L2 multimodal writing

This paper proposes a methodological framework for annotating semiotic resources and analysing the variations of their meanings in a corpus of second language (L2) disciplinary student writing. Most previous studies on semiotic resources or their meanings in student writing have relied on qualitative coding, which may limit the number of texts that could be analysed. This study proposes a quantitatively driven framework for analysing semantic domains of semiotic resources and uses a balanced corpus of 100 successful multimodal texts written by L2 student writers at the postgraduate level in UK higher education. Through collaboration between a locally run open-source vision language model and a human annotator, semiotic resources were annotated in the corpus, and then the meanings of these features were examined using semantic tagging and principal component analysis. The findings of the principal component analysis revealed three patterns of variation in the meanings of semiotic resources. One of the key findings is that the meanings of semiotic resources varied along a continuum, suggesting multifaceted rather than discrete meanings. This study’s annotation framework has implications for corpus construction and annotation in corpus studies of student writing that have largely overlooked multimodal features to date. Importantly, the study has methodological implications to examine the meanings and discourse functions of semiotic resources in future studies of L2 multimodal writing by using a mixed methods approach that moves from quantitative to qualitative analysis in a principled manner.