MEGAEL: automated multimodal event generation for IoT-driven process mining using generative AI

The proliferation of Internet of Things (IoT) devices in industrial environments generates vast heterogeneous data streams, yet extracting actionable process insights remains challenging due to the semantic gap between low-level sensor measurements and high-level business activities. Traditional process mining techniques assume the availability of structured event logs, whilst IoT environments produce raw multimodal data requiring extensive manual annotation to bridge this gap. Existing approaches suffer from three critical limitations: reliance on single modalities, poor semantic richness with minimal contextual attributes, and lack of spatio-temporal consistency validation. This paper introduces MEGAEL (Multimodal Event Generation using Adaptive Ensemble Learning), a comprehensive framework that orchestrates Large Language Models, Vision-Language Models, Retrieval-Augmented Generation, and hybrid Transformer-Graph Neural Network architectures for automated event log generation from heterogeneous IoT data. The framework operates through five synergistic phases: multimodal ingestion with temporal synchronisation, parallel LLM-based semantic extraction and vision-based contextual enrichment, knowledge-grounded augmentation using organisational documentation, and dual-branch spatio-temporal validation ensuring both temporal coherence and spatial consistency. Extensive evaluation on a real-world smart manufacturing dataset comprising 2.5 million sensor readings, video streams, and system logs demonstrates MEGAEL’s superiority over existing approaches. The framework achieves 0.91 F1-score, generates semantically rich event logs with 12+ attributes per event (versus 3-5 for baselines), reduces temporal violations by 74 % and spatial violations by 72 %, whilst maintaining near-real-time performance with 250-millisecond latency. These results confirm MEGAEL’s practical viability for operational process mining in IoT-enabled environments.