Preregistration: nortrebenk v1.0 pilot measurement (P3)

nortrebenk measures how well and how fast coding agents solve tasks taken from public code in Norwegian public agencies, and what each solved task costs. The tasks are merged pull requests from the agencies' public GitHub repositories. The ground truth is the agency's own tests: a task counts as solved when the tests that failed before the change pass, and the tests that passed still pass. Status: preregistered; the pilot measurement has not been run. The preregistration is frozen at git tag prereg-v1 (PREREG.md). No agent has been run on a task in the set. AI disclosure: implemented with Claude Code.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-17
DOI
https://doi.org/10.5281/zenodo.24304902
Citations
1
Primary Topic
Software Engineering Research
Type
article
Field-Weighted Citation Impact
2.46
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Preregistration: nortrebenk v1.0 pilot measurement (P3)

Eirik Botten Nicolaysen
1 citations
Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
2.46
article

Preregistration: nortrebenk v1.0 pilot measurement (P3)

Eirik Botten Nicolaysen
article en
1 citations

Abstract

nortrebenk measures how well and how fast coding agents solve tasks taken from public code in Norwegian public agencies, and what each solved task costs. The tasks are merged pull requests from the agencies' public GitHub repositories. The ground truth is the agency's own tests: a task counts as solved when the tests that failed before the change pass, and the tests that passed still pass. Status: preregistered; the pilot measurement has not been run. The preregistration is frozen at git tag prereg-v1 (PREREG.md). No agent has been run on a task in the set. AI disclosure: implemented with Claude Code.

Zenodo (CERN European Organization for Nuclear Research)
Openalex Percentile: Top 13%
Software Engineering Research
2.46
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon