Visual Parsing Algorithms for an Equitable Augmented Reality Learning System

Giving instructions for a character to navigate around a scene provides a simple analog for the planning needed in computer programming. While many children’s navigation games exist, most require the child to use a combination of input devices such as keyboard, mouse, and controllers for play. Children under the age of five may struggle to use a mouse, but they can easily construct the plans needed for such a game. This research explores layout and graph connectivity algorithms to connect tactile game pieces for a navigation game. A web camera identifies the position of action cards and numerical modifiers (card: go forward, modifier: 2 steps) on a table in front of the player. A suitable algorithm must be developed that is 1) robust to uneven placement, 2) can connect action in a sequence, and 3) connect modifiers to the appreciated action. While using tactile game pieces has a number of pedagogical advantages [1], allowing children to construct the navigation plan for controlling an on-screen character using optically recognized pieces eliminates the need for complicated input devices and lowers barriers for younger children. As the child places command cards and numerical modifiers to those commands on the table, their coordinates in space are relayed to the algorithm by a machine vision system. The current set of algorithms in development connects the commands and modifiers into a program representation that is then processed and animated when the young learner wants to run their program. These algorithms will allow for the unorthodox positioning that the children will discover through the course of play but still connect the programs in a way that captures the child’s intent. This is a work in progress report on the visual parsing algorithms of a larger learning system.