Beyond Single Metrics: A Holistic Benchmarking Framework for Low-Power Embedded Systems

Modern embedded systems encounter a notable challenge in evaluation. While devices may meet traditional benchmarks, they often underperform in real-world applications due to neglected interactions at the system level. Current benchmarking suites, such as MLPerf Tiny and EEMBC ULPMark, evaluate specific metrics including computational throughput, energy efficiency, and memory usage. However, they do not consider the complex interdependencies that affect real-world performance. This thesis presents a benchmarking framework that concurrently evaluates multiple performance dimensions under realistic workloads, revealing system behaviors that are often hidden in conventional benchmarks.Through the comprehensive evaluation of three representative algorithms: Fast Fourier Transform, quantized neural network inference, and Dijkstra's shortest-path algorithm, across 33 hardware platforms, including ARM Cortex-M microcontrollers, ultra-low-power FPGAs, and edge AI accelerators. By integrating cycle-accurate hardware models with RTOS scheduling, power state transitions, and I/O emulation, our framework uncovers critical system-level effects: memory requirements 2.5×-9.6× higher than algorithm size alone (averaging 8.4×), performance degradation up to 42.8% from I/O interference, and power state transition overheads that can dominate energy consumption at low duty cycles. Our simulation-based analysis identifies previously overlooked factors, including bus contention between CPU and DMA operations, interrupt-induced cache pollution, and priority inversion penalties, which collectively impact performance by 20-40%. We present the following contributions: (1) a cohesive measurement architecture that links previously isolated metrics, (2) quantitative evidence revealing systematic measurement errors in current benchmarks, and (3) empirically derived design guidelines that incorporate 2-3× memory safety margins and platform selection criteria informed by observed system behavior. This research allows precise performance forecasting for resource-limited systems, shifting embedded system design from intuition-based approaches to data-driven optimization.