Semantic prosody, categorisation and inter-rater reliability

This article investigates the inter-rater reliability of established methods of categorising semantic prosody. Semantic prosody is a concept associated with corpus linguistics, which describes the tendency of seemingly neutral items to occur in particular evaluative contexts. In previous research on semantic prosody, there has been a heavy reliance on manual analysis of smaller samples, and because of this, questions have been raised about the stability of the established methods for categorisation. Furthermore, there is also a lack of consensus regarding how such categorisations should be operationalised. Traditionally, it has often been viewed in binary terms, distinguishing between positive and negative prosodies. However, this restricted system has also received criticism, and certain researchers have adopted a more comprehensive (or fine-grained) categorisation, more connected to a unit’s semantic preference. This paper aims to evaluate the inter-analyst consistency of these systems through two experimental studies, in which four researchers independently analyse the same set of random concordance lines of the items habit and views from BNC2014, applying both methods of categorisation. The results indicate that a binary distinction between positive and negative offers a higher inter-analyst consistency than a more fine-grained categorisation. Additionally, this more comprehensive system was also found to obscure the borders between semantic preference and semantic prosody. However, because neither system achieved satisfactory inter-rater agreement, both studies highlight the need for more objective methods of analysing and categorising semantic prosody.