Re-examining ASSERT: An Open-Model Replication and a Scenario-Level Re-analysis of Released Outputs

ASSERT [8] tests the robustness of language models on physical-safety questions fromSafeText [6] with three automatic red-teaming methods: semantically aligned augmentation(SAA), targeted bootstrapping (TB) and adversarial knowledge injection (AKI). It compareseach method against the original benchmark with a two-proportion z-test. We make twocontributions. First, we replicate all three methods with an open 4-bit model, Qwen2.5-7BInstruct-AWQ [10], on a free Colab T4 GPU, with pinned code, data and weights. Second,we re-analyse the outputs the ASSERT authors released for GPT-3.5, GPT-4, Alpaca andVicuna. The released paraphrases do not record their source scenario, so we recover thelink with an order-preserving aligner that is exact on unsafe items in simulation. From thereleased files we reproduce all 16 overall effects in the paper’s Tables 2–4 to within 0.03points, the precision of its printed values. We then compare each method with the baseline onthe same scenarios and resample whole scenarios rather than single items. AKI’s large effectsurvives in every setting, and in our replication it raises Qwen’s error on unsafe scenariosfrom 4.6% to 24.6% (+20.0 points, 95% CI [16.1, 24.0]). Two published conclusions change.Vicuna’s significant SAA effect on unsafe scenarios (−7.27 accuracy points, p < .01) shrinksto +2.7 error points (95% CI [−0.8,6.2]) once Vicuna is compared on the 245 of 370 scenariosfor which it produced paraphrases. GPT-3.5’s non-significant SAA effect on unsafe scenarios(p = 0.23) becomes borderline significant (p = 0.04) under a paired analysis. We also findthat the change in error reported for cross-model attacks appears to be computed againstthe source model’s baseline rather than the target’s. We release code, the run manifest andall aggregate results.