Gene position and relatedness confound genomic language model benchmarks

Benchmark scores of genomic language models are read as evidence of learned genome organisation and gene function; we ask whether gene position and genome relatedness explain them instead. Across six benchmarks and two label sets built without genome layout, we compare models with baselines that use no sequence model and, where possible, score them with position held fixed. The credit that models receive depends on how a benchmark's labels, inputs and splits were built: where these encode layout or relatedness, much of that credit belongs to information available without a model. Position alone beats every language model compared on bacterial operon benchmarks, as in a companion study. On most eukaryotic long-range tasks, distance matches or beats most published DNA language models. On labels built without layout, no tested model gains more than 0.002 AUROC over position on operons labelled from nanopore reads of expressed genes, and little model signal on experiment-only interaction labels survives strict controls. Relatedness explains much of the scores on BacBench's two genome-level tasks, and in further DNA benchmarks, train–test overlap or negatives built from positives let a rule or lookup beat the best published score we found on 24 of 81 tasks. Real signal remains on gene-level tasks, where it comes from the identity of genes. Signal beyond position also remains on STRING interaction labels and in fine-tuned DNA language models on a causal eQTL task, where it is often not specifically genomic. Neither larger models nor the pretraining changes tested remove the position confound. Benchmarks therefore need position and relatedness baselines, metrics with position held fixed and labels built without layout; we list a fix for each defect and release seqleak, which computes the position baseline and these metrics.