Agentic Reverse Engineering of Embedded AI Models: A Benchmark Design and Evaluation Protocol

LLM agents can now drive reverse-engineering tools such as Ghidra and IDA Pro through MCP servers, but how well they find and recover AI models embedded in mobile apps and firmware has not been measured. This paper reports no experiments. It proposes a benchmark design and an evaluation protocol for that measurement, from the defender’s side. The benchmark consists of author-built applications that embed known open-source models in several formats under graded protection levels, from plain assets to encrypted weights loaded by native code, so every future result will have exact ground truth. The protocol evaluates a tool-using agent against baselines on five stages: discovery, format identification, access, architecture recovery, and ancillary artifacts such as embedded prompts and labels. It also defines metrics that separate an honest failure from a false claim, a failure taxonomy, and a test of whether content planted in an analysed binary can steer the agent. We state the hypotheses to be tested and the rule for turning results into guidance on which protections slow an LLM agent.