Indonesian Automated Short Answer Scoring Using Sentence Transformers and Siamese LSTM

The objective of automated short-answer scoring is to provide automated scores that are as close as possible to human scores. However, the lexicon-based approach using a bag of words has not yielded satisfactory score predictions due to the possibility that sentences with identical meanings can be conveyed using different words. To address the limitation, this study proposes the use of semantic features derived from pretrained Sentence BERT (SBERT) embedding vectors, which are then fed into the Siamese LSTM. The proposed model was evaluated using an Indonesian automated short answer scoring benchmark dataset. Furthermore, cosine similarity and Manhattan distance were evaluated as output functions on the Siamese LSTM. The experimental results obtained from the test data yielded an optimal Pearson’s correlation of 0.920 and a Root Mean Squared Error (RMSE) of $\mathbf{1 0. 0 1 9}$ when SBERT and Siamese LSTM were fine-tuned using Manhattan distance as the output function. These values show a correlation of 0.77 and an RMSE of 10.019, which are 0.77 and 7.154 higher, respectively, than the values reported in the previous study. The test result for cosine similarity as the output function yielded the optimal RMSE of 11.494 and correlation of 0.892, which represent improvements of 5.679 and 0.049, respectively, in RMSE and correlation, respectively, over the previous study. These results illustrate that the integration of sentence embeddings and LSTM in this study has produced superior results in Indonesian automated short answer scoring on the benchmark dataset using lexicon-based techniques.