JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse

Recently, action-based decision-making in open-world environments has gained significant attention.Visual Language Action (VLA) models, pretrained on large-scale web datasets, have shown promise in decision-making tasks.However, previous work has primarily focused on action post-training, often neglecting enhancements to the foundation model itself.In response, we introduce Act from Visual Language Post-Training (ActVLP), a novel training paradigm.ActVLP distinctively enhances the foundation model prior to action-specific tuning by first post-training it on a curated set of environment-specific visual and linguistic tasks using self-supervised learning.This initial stage significantly improves the model's capabilities in world knowledge, visual recognition, and spatial grounding.Subsequently, this strengthened VLM undergoes action posttraining via imitation learning on trajectory datasets.Following this paradigm, we develop JARVIS-VLA, the first VLA model in Minecraft that can follow human instructions on over 1k different atomic tasks, including crafting, smelting, cooking, mining, and killing.Our experiments demonstrate that our ActVLP paradigm leads to a significant 40% improvement over the best agent baseline on a diverse set of atomic tasks.Furthermore, JARVIS-VLA surpasses traditional imitation learning-based policies in Minecraft, achieving state-of-the-art performance.We have open-sourced the code, models, and datasets to foster further research.The project page can be found at https://craftjarvis. github.io/JarvisVLA.