Scan pattern similarity predicts the semantic similarity of sentences across languages above and beyond their syntactic structures.

Human cognition is a highly integrated system which synchronizes processes and representations across modalities. Our previous research on the synchronization between overt attention and human sentence production demonstrated that similar scene descriptions correspond to similar sequences of objects being attended to (i.e., scan patterns). Here, we generalise this finding from English to languages with a different word order. More specifically, we test whether synchronicity holds not just within a given language but across languages and examine the relative contribution of syntax and semantics. 74 participants (24 British English, 28 European Portuguese and 20 Japanese) were asked to describe an object (N = 24), either animate (e.g., man) or inanimate (e.g., suitcase), situated in a visual scene, and prompted with a cue word, while being eye-tracked. Across all participants, pair-wise similarities of sentences were computed using the Universal Sentence Encoder, which generates vector-based meaning representations across languages. Part-of-Speech (POS) sequences, which are a shallow representation of the syntax of sentences, were extracted using Spacy. Similarities between POS sequences and scan patterns (i.e., sequences of fixated objects) were computed using the Longest Common Subsequence algorithm. First, we successfully replicated that similar sentences are associated with similar scan patterns in all three languages. Moreover, we demonstrated for the first time that this relationship holds across languages: for instance if a Japanese and a Portuguese sentence are semantically similar, their associated scan patterns are also similar. In contrast to semantic similarity, we find that syntactic (POS) similarity is predicted by scan patterns only within the same scene, and only between languages with similar word order. This result not only confirms that visual attention and language production are synchronized, but it theoretically points at a grammar of perception that is universal across languages, goes beyond (syntactic) surface realizations, and empirically manifests as oculomotor responses.