Instruction Following Using Vision and Natural Language

Embodied instruction following is a complex task that involves an agent inferring a series of basic actions from complex language and visual inputs in order to achieve a desired state in the environment. To address this challenge, a benchmark called Action Learning From Realistic Environments and Directives (ALFRED) has been introduced. ALFRED presents a collection of detailed natural language instructions that guide the agent in accomplishing subgoals that collectively contribute to achieving a higher-level objective. ALFRED dataset [1] contains roughly 25K language annotations of both high-level goals and low-level step-by-step instructions for various long-horizon and compositional tasks set in the AI2-THOR simulator [2]. Key challenges for this task include localizing target locations and navigating to them through visual inputs and grounding language instructions to the visual appearance of objects. In this project, we have provided results for an end-to-end approach that are better than the baselines in all the metrics but cannot beat the current state-of-the-art [3]. We have also worked on modular approaches to solve such tasks. We modify the deterministic policy in the existing modular approach FILM [4] and improved the performance by 5.82%. We also considered creating semantic maps [5] by matching photos to descriptions of item purposes in natural language that allows us to transform the navigation description of the goal into a list of open-vocabulary navigational objectives that are immediately localized on the map.