Computational elements of natural vision

Ultimately, human behavior needs to be understood in the context of natural everyday tasks. Over the last two decades, a number of observations of natural visually guided behavior have accumulated. These observations help define the functional demands placed on the visual system in a variety of tasks, but progress has been limited by the diversity of natural behavior and by the lack of unified theoretical structures to guide understanding of the underlying processes. In this article, we summarize some recent attempts that might provide a template for a more formal approach. This is possible because it has become clear that natural behavior has many regularities reflecting the underlying sensorimotor decisions. We first summarize these regularities and then show how simple visually guided behaviors can be well described by partially observable Markov decision processes. We give examples of how laboratory experiments can be designed to elicit the common elements of natural behavior and how such experiments afford control of the statistical structure of tasks, thereby allowing formal modeling. Finally, we suggest that a new exciting avenue using recently introduced inverse models may lead the way forward, as it recovers the intrinsic properties of human perception, cognition, and action, which are intertwined in natural behavior.