STP-H12-VANTAGE: Combining Visual and Event-Based Sensing for Earth Observation

Conducting Earth-observation tasks from orbit can provide valuable research insights in fields such as meteorology, geology, forestry, cartography, and defense. Developing and building space payloads for Earth observation poses significant challenges, including the high cost of radiation-tolerant systems, the widening gap between current state-of-the-art apps and the capabilities of space-grade processors, and the increase in fidelity of modern sensors. Many Earth-observation missions typically collect data by leveraging visual and/or multispectral imagers. However, when using conventional imaging, there can be significant losses in context due to the frame-based operation of these sensors, as well as both relevant and irrelevant data mixed together in one frame. Additionally, high-resolution visual sensors can produce large amounts of data, which poses problems for onboard storage and processing. Event-based dynamic vision sensors, also known as neuromorphic sensors, capture only changes in light intensity throughout a scene. These sensors only react to events or “differences” in a scene, enabling only the relevant information from a scene to be extracted and providing a much lower minimum data rate, all while drawing less power. Neuromorphic sensing has been successfully deployed in orbit, and neuromorphic sensors have shown resilience to space radiation in prior research. This paper introduces the Visual And Neuromorphic Tracking And Geosensing Experiment (VANTAGE), which seeks to intelligently combine visual imagery and neuromorphic event-based sensing. These sensors will operate in tandem with an overlapping field of view to further enable onboard autonomy. Onboard data collection will also be highly energy efficient, as the low-power neuromorphic sensor will be used as an intelligent trigger to capture images with the higher-power camera. By overlapping the field of view of its onboard sensors, VANTAGE will be able to simultaneously collect visual and event-based information detailing subjects on Earth, building a set of correlated data between the two sensing technologies. Furthermore, VANTAGE will leverage several hybrid (incorporating commercial and radiation-tolerant), low-cost, high-performance processing systems. At the core of the VANTAGE payload is a fault-tolerant single-board computer introduced in previous missions. This computer is connected to additional processing systems, which consist of commercial system-on-module-based processors paired with fault-tolerant carrier cards. The result is a highly diverse and fully autonomous flight system capable of conducting science with advanced sensing, processing, and efficient data downlink capabilities. Ultimately, this payload will leverage its sensing and processing suite for a variety of novel imaging applications. VANTAGE is currently in development and has been manifested for flight on the Space Test Program’s STP-H12 mission.