A State‐Aware Fuzzing Method for Trusted Execution Environment Kernels

ABSTRACT Trusted execution environments (TEEs) are widely used, and their kernel security has become a significant area of focus. Fuzzing, a powerful technique for detecting vulnerabilities in operating systems, has increasingly been applied to the security analysis of TEEs. However, conventional fuzzing tools cannot be directly used for TEE kernels due to their isolation. Coverage‐guided fuzzers often discard test cases that trigger new states but cover the same code, which limits their effectiveness in discovering vulnerabilities. To address these challenges, we propose a state‐aware fuzzing method specifically designed for TEE kernels. Initially, we develop a modeling and tracing approach to represent the program state through state‐variable values, overcoming the limitations of coverage‐guided fuzzers. Subsequently, we propose a new communication scheme to address the issues resulting from the isolation of TEEs. Additionally, new seed preservation and selection algorithms are put forward to better guide the fuzzer in exploring vulnerabilities. Finally, we employ the N‐gram model to enhance the test case generation process and optimize the framework's performance. We have implemented a prototype called Trusty‐Statefuzz and evaluated it on Fuchsia, our self‐developed microkernel operating system Nebula, and the OP‐TEE. The evaluation results show that Trusty‐Statefuzz is effective at detecting both new code and vulnerabilities. Trusty‐Statefuzz discovers nine unknown vulnerabilities and 23 known vulnerabilities. Additionally, it achieves 13% higher code coverage and 27% higher state coverage than the state‐of‐the‐art fuzzer Syzkaller.