Evaluation of the phi-3-mini SLM for Identification of Texts Related to Medicine, Health, and Sports Injuries

Small Language Models (SLMs) have potential to be used for automatically labelling and identifying aspects of text data for medicine/health-related purposes from documents and the web. As their resource requirements are significantly lower than Large Language Models (LLMs), these can be deployed potentially on more types of devices. SLMs often are benchmarked on health/medicine-related tasks, such as MedQA, although performance on these can vary especially depending on the size of the model in terms of number of parameters. Furthermore, these test results may not necessarily reflect real-world performance regarding the automatic labelling or identification of texts in documents and the web. As a result, we compared topicrelatedness scores from Microsoft's phi-3-mini-4k-instruct SLM to the topic-relatedness scores from 7 human evaluators on 1144 samples of medical/health-related texts and 1117 samples of sports injury-related texts. These texts were from a larger dataset of about 9 million news headlines, each of which were processed and assigned scores by phi-3-mini- 4 k-instruct. Our sample was selected (filtered) based on 1 (low filtering) or more (high filtering) Boolean conditions on the phi-3 SLM scores. We found lowmoderate significant correlations between the scores from the SLM and human evaluators for sports injury texts with low filtering (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\rho=0.3413, \mathrm{p}&lt;0.001$</tex>) and medicine/health texts with high filtering (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\rho=0.3854, \mathrm{p}&lt;0.001$</tex>), and low significant correlation for medicine/health texts with low filtering (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\rho=0.2255, \mathrm{p}&lt;0.001$</tex>). There was negligible, insignificant correlation for sports injuryrelated texts with high filtering (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\boldsymbol{\rho}=\mathbf{0. 0 3 1 8}, \boldsymbol{\text{ p }}=\mathbf{0. 4 4 6 6}$</tex>). (Abstract)