DamageRAG: Benchmarking and diagnosing multi-hop retrieval-augmented reasoning for structural damage inspection

Structural damage inspection requires cross-image retrieval and multi-hop reasoning that current multimodal large language models (MLLMs) and retrieval-augmented generation (RAG) pipelines do not deliver. We contribute a benchmark, a pipeline whose structure makes failures attributable, and the diagnosis that attribution supports. The benchmark (dacl10k v2) has 5794 multiple-choice questions over 7910 images, 11 task types and 3 hop levels, with deterministic ground truth from polygon annotations. DamageRAG retrieves over MLLM-generated structured Damage Reports by field matching and routes quantitative queries to programmatic solvers; no stage is trained. Given ground-truth Reports it lifts a 7B model from 57.9% to 82.9%. A three-layer diagnosis separates retrieval, reasoning and perception failures; perception dominates: under MLLM-generated Reports the full pipeline falls to 29.5% against a 25.0% random floor, 3.0 pp above a no-retrieval baseline ( p = 0.020 ) and not separable from the strongest naive RAG baseline (31.0%, p = 0.249 ). A perception-side sensitivity analysis shows the bottleneck is not architectural: LoRA fine-tuning the perception model on in-domain images raises macro F1 from 0.37 to 0.60 and accuracy to 37.5%, is worth 0.0 pp to a CLIP-retrieval baseline, and, with every system given comparable-budget adaptation, moves the pipeline from 1.5 pp behind the strongest baseline to 6.5 pp ahead ( p < 1 0 − 6 ). Two scope limits apply: recovering query structure from free-form phrasing drops the quantitative branch from 79.9% to 41.9%–46.4%, and the router misroutes 9.6% of template queries at 50.4 pp each. On this benchmark, open-vocabulary damage perception, not retrieval or reasoning, separates current MLLMs from a deployable inspection assistant.