EventSleep2: Sleep activity recognition on complete night sleep recordings with an event camera

Sleep is fundamental to health, and society is more and more aware of the impact and relevance of sleep disorders. Traditional diagnostic methods, like polysomnography, are intrusive and resource-intensive. Instead, research is focusing on developing novel, less intrusive or portable methods that combine intelligent sensors with activity recognition for diagnosis support and scoring. Event cameras offer a promising alternative for automated, in-home sleep activity recognition due to their excellent low-light performance and low power consumption. This work introduces EventSleep2-data , a significant extension to the EventSleep dataset, featuring 10 complete night recordings (around 7 h each) of volunteers sleeping in their homes. Unlike the original short and controlled recordings, this new dataset captures natural, full-night sleep sessions under realistic conditions. This new data incorporates challenging real-world scene variations, an efficient movement-triggered sparse data recording pipeline, and synchronized 2-channel EEG data for a subset of recordings. We also present EventSleep2-net , a novel event-based sleep activity recognition approach with a dual-head architecture to simultaneously analyze motion classes and static poses. The model is specifically designed to handle the motion-triggered, sparse nature of complete night recordings. Unlike the original EventSleep architecture, EventSleep2-net can predict both movement and static poses even during long periods with no events. We demonstrate state-of-the-art performance on both EventSleep1-data, the original dataset, and EventSleep2-data, with comprehensive ablation studies validating our design decisions. Together, EventSleep2-data and EventSleep2-net overcome the limitations of the previous setup and enable continuous, full-night analysis for real-world sleep monitoring, significantly advancing the potential of event-based vision for sleep disorder studies. • EventSleep2-data, an extension of the EventSleep dataset (EventSleep1-data), that includes new event camera sleep recordings. These recordings capture 10 volunteers during their night sleep, around 7-hour long. Six of these recordings are synchronized with EEG data captured from a neuroheadband. This length and additional sensor data stored required a new event-triggered recording protocol. EventSleep2-data provides unique new data for the sleep research community, in particular for the promising application of event camera data for health studies in this field. • A new sleep activity recognition approach (EventSleep2-net) based on event camera data. This approach is built on the original EventSleep approach (EventSleep1-net), but it is more robust and better suited to work with complete night sleep recordings. These advantages are demonstrated with superior performance than previous work on both versions of the EventSleep and EventSleep2 data. • A discussion of the EEG-event data relationship.