Ecological semiotics: Multimodality, multilingualism, and situated language learning in the AI era

Using a variety of media beyond text—audio, video, visual—has long been a common practice in second language learning. Digital media has dramatically increased that opportunity. Multimedia-friendly mobile devices provide access to all media formats, leading to the frequent practice of combining modes. At the same time, social media has increasingly seen a mixing of languages along with the use of images, voice, and video. Studies of multimodal enhancements in second language acquisition (SLA) have generally shown positive learning outcomes, although results vary widely depending on conditions. A variety of theories have been developed in support of the mediating and motivating effects of integrating multimedia into language instruction. Those include different theories of semiotics, the science of signs, as well as the use of ecological theories such as sociomaterialism. The use of learners’ first language (L1) in instructed SLA has been shown to be a valuable scaffolding tool (often in combination with multimodality), leading to the practice of pedagogical translanguaging. Since 2022, the availability of generative artificial intelligence (AI) has been shifting the media landscape. As AI systems steadily improve in multimedia capabilities, existing methods and theories on multimedia integration into SLA may need to be revisited. An approach based on ecological semiotics may prove helpful.