Validating event-based vision sensor characterization methods through simulation

Event-based Vision Sensors (EVS) represent a relatively new class of vision sensor that achieves key advantages over conventional frame-based cameras. Each EVS pixel operates independently of its neighbors, generating output ”events” only when its incident light level changes, to produce a data-sparse representation of dynamic information in a visual scene with low latency and high temporal resolution with low power and across a wide Dynamic Range (DR). Despite many advantages, characterizing and describing EVS performance with practical metrics for consumers continues to be a challenge. Methods and metrics used to characterize conventional cameras are not compatible with their unique mode of operation. Performance metrics needed to describe EVS sensors include thresholds (the minimum log-photocurrent change needed to trigger a pixel ”event”), pixel bandwidth or temporal response (how fast the pixel can respond to changes), refractory period (a programmed ”dead time” during which a pixel ignores changes after reporting an event), and noise (typically measured in terms of false event rates). Accurately measuring any one of these performance metrics is a nontrivial task, and many of these parameters change (or in some cases appear to change) as a function of illumination. Throughout the past decade, several researchers have laid the groundwork for EVS characterization methods, and the resulting metrics are often reported as a means to quantitatively compare performance of both commercial and research EVS prototypes. Despite this, the community still lacks a standard testing protocol, and the accuracy of reported metrics greatly depends on the method, testing environment, and interpretation. Until now, no study has attempted to validate EVS test methodologies or quantify their ability to produce accurate estimates of each parameter. This study employs a realistic pixel-level simulation to compare various testing methods used to characterize EVS and quantify their accuracy. Additionally, best practices are identified along with more accurate ways to interpret data. Two independent measurement techniques are shown to reliably estimate thresholds to within 2% relative error, although one is more compatible with low-light measurements. Among the pixel bandwidth measurement methods evaluated, we find that the conventionally used method is unreliable in accurately predicting Low-Pass Filter (LPF) corner frequency, and we propose a new method that we demonstrate to reliably estimate frequency response to ≤ 10% error up to 5kHz when the refractory period is 10μs. We also provide insight into a previous method for measuring refractory period, and show that it can estimate refractory periods as low as 20μs to within 10% error with pixel bandwidth ≥ 5kHz. Finally, we include a discussion on noise characterization, and propose development of a more complete metric which accounts for threshold, bandwidth, and refractory period.