MEE: A Multimodal Affective State Estimation Engine for AI Companions and Intelligent Robots

Human interaction is inherently multimodal. People communicate not only through language, but also through facial expressions, vocal characteristics, temporal patterns, and other observable behavioral signals. Conventional conversational AI systems, however, frequently rely primarily on textual or speech input and therefore have limited access to non-linguistic interaction context. This technical note presents MEE (Multimodal Emotion Engine), a modular multimodal affective-state estimation system designed for AI companions, intelligent robots, and other human-AI interaction systems. MEE combines facial-expression estimation and voice-based emotion estimation through a deterministic multimodal fusion architecture. The system produces structured affective-state information including estimated emotion, confidence, modality-specific scores, multimodal conflict, temporal trend, and processing latency. A central design principle of MEE is the separation of perception from decision-making. MEE estimates observable affective-state signals; it does not determine what an AI companion or robot should do in response. Behavioral decisions remain the responsibility of the higher-level companion or robotic system. The current architecture provides a Python API centered on an EmotionEngine abstraction, configurable modality processors, deterministic fusion, temporal state tracking, and structured JSON-compatible output. The architecture is designed to support local and edge deployment, including Jetson-class hardware, while maintaining replaceable model adapters. The system is intended to serve as a perception layer within a larger AI Companion architecture: Sensors ↓ MEE ↓ Affective State ↓ Companion Context ↓ Memory / Reasoning / LLM ↓ Behavior ↓ Robot MEE is therefore not presented as a psychological inference system. Its purpose is to estimate observable affective expression from available multimodal signals and provide structured contextual information to downstream intelligent systems. Keywords: multimodal AI; affective computing; emotion estimation; facial expression estimation; voice emotion estimation; AI companion; intelligent robots; edge AI; embodied AI; multimodal fusion; human-AI interaction