From Games to Understanding: Semantrix as a Testbed for Advancing Semantics in Human–Computer Interaction with Transformers

Despite rapid progress in natural language processing, current interactive AI systems continue to struggle with interpreting ambiguous, idiomatic, and contextually rich human language, a barrier to natural human–computer interaction. Many deployed applications, such as language games or educational tools, showcase surface-level adaptation but do not systematically probe or advance the deeper semantic understanding of user intent in open-ended, creative settings. In this paper, we present Semantrix, a web-based semantic word-guessing platform, not merely as a game but as a living testbed for evaluating and extending the semantic capabilities of state-of-the-art Transformer models in human-facing contexts. Semantrix challenges models to both assess the nuanced meaning of user guesses and generate dynamic, context-sensitive hints in real time, exposing the system to the diversity, ambiguity, and unpredictability of genuine human interaction. To empirically investigate how advanced semantic representations and adaptive language feedback affect user experience, we conducted a preregistered 2 × 2 factorial study (N = 42), independently manipulating embedding depth (Transformers vs. Word2Vec) and feedback adaptivity (dynamic hints vs. minimal feedback). Our findings revealed that only the combination of Transformer-based semantic modelling and adaptive hint generation sustained user engagement, motivation, and enjoyment; conditions lacking either component led to pronounced attrition, highlighting the limitations of shallow or static approaches. Beyond benchmarking game performance, we argue that the methodologies applied in platforms like Semantrix are helpful for improving machine understanding of natural language, paving the way for more robust, intuitive, and human-aligned AI approaches.