Viveka: Separating Real Gains from Chance in Self-Improving Agents

Agents that improve their own harness usually keep a change when a score goes up. That is weak evidence. Agent evaluations are noisy, and an agent that tries many changes will keep some that only looked better by chance. We show two simple facts. Under the usual "keep it if it scores higher" rule, a change that does nothing is kept half the time, no matter how many seeds are used. And once such a change is kept, its lucky score becomes the bar that real improvements must clear, so real progress slows down. We propose Viveka, a checking layer that sits outside the agent. It re-tests each change on fresh seeds the agent has never seen, stops early when the answer is clear, replaces lucky scores with fresh ones, and later removes changes that turn out not to matter. In a simple simulation of 100 proposed changes, the usual rule reports a gain of 1.92 sigma when the real gain is 0.52 sigma. Viveka reaches a real gain of 1.43 sigma, while spending a similar budget on extra seeds reaches 0.82 sigma. These numbers come from a simulation, not from real agents. We end with a fixed plan for testing the same claims on an open-source self-refining coding agent.