Open-Source or Proprietary Language Models? An Initial Comparison on the Assessment of an Educational Task

Open-source Small and medium Language Models (SLMs) are becoming more and more available, and allow researchers to fine tune and explore their application with more freedom than the proprietary ones. We compare the performances of many recent SLMs with two proprietary models from OpenAI, on two didactic tasks regarding the automatic assessment of algorithm descriptions submitted for an under-graduate course, involving training on Computer Programming. In particular, as the submission is part of a formative peer-assessment workflow, where peers grade three descriptions each, and suggest improvements, we are interested in two tasks: a) discriminating non-algorithms descriptions from algorithm descriptions, and b) grading the clearness of the submitted descriptions (which is related to an effective usefulness during peer-assessment). We see that the tasks are solved with varied performances by the tested Models, and that a proprietary Model can be leader on a task, whereas some SLMs are better at the other task. Then we also show some strategies, to improve the grading/classifying performances. The development of Language Models that are Open Source, Open trained, and overall smaller than the available proprietary ones, is currently at an early stage: nonetheless a conclusion can be that these systems may already have their say, when compared with large, proprietary Language Models.