AIGLLM: An Action Instruction Generation Method with Visually Enhanced LLM

Action Instruction Generation (AIG) is a key step in semantic-driven action sequence generation, which aims to embed text instructions describing specific actions into the driving text. AIG is a challenging task that requires accurately understanding the potential action semantics contained in the driving text. Besides, the challenge is exacerbated by the lack of paired text-action instructions data. To address these challenges and motivated by the powerful capabilities of LLMs in text generation, we propose AIGLLM, a novel AIG framework with visually enhanced Large Language Model (LLM). Specifically, we perform low-rank adaptation (LoRA) on the off-the-shelf LLM to enable the model understand action semantics and generate action instructions at appropriate positions within the original text sequence. Particularly, to address the current lack of data that can be used to fine-tune LLMs for AIG, we construct a dataset containing 12,000 text-action instruction pairs and 15,000 image-action instruction pairs. Subsequently, a visual module is designed to enable the LLM to learn action knowledge from external visual signals of actions, thereby enhancing the model's understanding of action semantics. Finally, we conduct extensive experiments, where the experimental results and evaluations demonstrate that our method has an impressive key benefit: improved accurate action instruction embedding.