Seeing Through Draw Calls: Learning Object Semantics from a Game's Render Commands

AI systems that need to understand a game or 3D application today look at its pixels. Every frame, though, is built from draw calls: instructions to the graphics card that carry each object’s geometry, transform, depth and material before any pixel exists. This research brief proposes learning object semantics directly from that stream. Draw calls are grouped into object candidates, each described by sampled geometry, material descriptors and scene context, and a small set encoder classifies objects, detects scene changes and flags rendering bugs. The same object tokens can also be passed to an existing vision-language model alongside, or instead of, a screenshot. The study tests whether scene context adds to geometry alone, how far it transfers to unseen assets, games and engines, and whether the tokens improve an existing model’s answers about a scene.