CTI-Rationale: An Expert-Annotated Dataset and Evaluation Framework for Evidence-Grounded MITRE ATT&CK Technique Mapping

Mapping cyber-threat intelligence (CTI) to MITRE ATT&CK is usually evaluated as label prediction, yet a correct technique identifier may rest on the wrong words. We treat each mapping as an argument (a technique claim supported by textual data, a warrant, and rejected alternatives) and annotation as discovery followed by judgment. CTI-Rationale instantiates this model with 1740 expert paragraph–technique mappings from 100 public reports, each with an evidence span, free-text rationale, reasoning type, and confidence, and most with primitives and ruled-out alternatives. All rationale content was written and adjudicated by human analysts, not generated by artificial intelligence. The low paragraph-level Jaccard (0.421) is associated mainly with coverage differences, not conflicting labels: paragraphs mapped by both annotators reach 0.780, only 10.4% of one-sided mappings are conflicts, and capture–recapture estimates put gold completeness at 84.2–88.0% (likely optimistic), lowest for behavioral and contextual warrants. In cross-validation, the annotation bundle improves micro-F1 beyond an equal-volume control (+0.024 [0.013, 0.035]), also under campaign-grouped and publisher-held-out folds and in document-disjoint transfer to TRAM2; ATT&CK definition text recovers about two thirds of the gain over label-only training. Evidence-supervised span ranking improves overlap with expert evidence, which label metrics do not capture. With fully delivered prompts, retrieved demonstrations raise a smaller language model’s micro-F1 from 0.367 to 0.500 (labels) and 0.475 (full annotations; no reliable added benefit). Unlabeled chunks remain unverified candidate negatives.