ANEForge: Python for Direct Computation on the Apple Neural Engine

ABSTRACT Background The Apple Neural Engine (ANE) is the fixed‐function neural accelerator on every recent Apple device, and the only sanctioned route to it is CoreML. CoreML treats the engine as a scheduling option: no configuration requires it, the CPU remains a permitted fallback, and no runtime interface reports which unit ran a call. Objective We set out to program the ANE directly from Python, with the compute unit fixed so that each measurement belongs to the engine, and to report what the engine computes and at what cost. Methods ANEForge compiles a lazy tensor graph into one ANE program and dispatches it through Espresso, the private runtime that CoreML is built on. It needs no entitlement and leaves system‐integrity protection on. The package reaches the engine's native fused‐attention layer, streams int8, int4, and sparse weights, keeps decoder and optimizer state on the engine across calls, runs the forward pass, backward pass, and optimizer update of training on the engine, and compiles fixed‐iteration Krylov solvers and a factored Fourier transform as static‐dataflow programs. We measured latency, energy per inference, and agreement with framework references on an M5 Pro, against the GPU and CPU at matched precision. Results A pretrained ResNet‐18 forward runs in 0.33 ms and uses 2.3 mJ per inference, about a factor of 9 less than the GPU at matched fp16 precision (paired runs give 5.5 to 12.7). A call costs about 74 μs whether the program fuses one operator or 128. ResNet‐18, a Vision Transformer, and a sentence encoder each compile to one program and match their framework references at cosine similarity 1.0000. One forward pass of the Stable Diffusion U‐Net and VAE decoder agrees with the reference to a relative error of 0.041 and 0.021. Conclusions The engine's advantage comes from fusion. A single memory‐bound operator finishes sooner on the CPU or the GPU, while a fused program pays the per‐call floor once. Fixed‐iteration methods whose program size does not depend on the problem dimension compile to the engine, and half precision limits how ill‐conditioned a problem they can solve.