From Artificial to Emergent Intelligence: Five Testable Criteria and a First Pre-registered Profile

Emergent Intelligence, Defined So It Can Fail: A Pre-registered Test in AI Agents That Invent Their Own Language Amit Brahmbhatt · Quantum Clarity LLC, Milpitas, California, USA · ORCID: 0009-0003-5882-8476 Preprint, version 1.0 (revised text) · 31 October 2026 · DOI: 10.5281/zenodo.23849135 Abstract “Emergent” is used loosely for capabilities that nobody designed. We propose an operational definition that can fail. A capability is emergent with respect to its designer specification to the degree that it meets five criteria: it goes beyond the training objective (C1), keeps developing after convergence (C2), is caused by interaction at equal compute (C3), comes with self-organized structure above an untrained baseline (C4), and, if it appears to switch on with scale, survives continuous metrics (C5). A system receives an EI profile of PASS/FAIL/INCONCLUSIVE verdicts rather than a label. We implement the criteria in EI-Bench, an open harness, and apply it to populations of neural agents that invent a discrete language in a reconstruction game. Every threshold was pre-registered in a public git history before the runs that test it. Across three rounds and 256 runs on one GPU, the profile is: C1 PASS (held-out accuracy 0.90–0.99 on unseen attribute combinations); C2 PASS, observed in all three rounds; C3 FAIL, including with heterogeneous populations; C4 FAIL at 20 seeds; and the apparent capability jump with model size is a metric MIRAGE (per-attribute transition 2.6× wider than the exact-match transition). Here C1 tests generalization to unseen combinations within the trained task. Turnover also outperformed a shrink-and-perturb control designed to restore plasticity while retaining learned weight directions: the difference in held-out accuracy gain was +0.19 (95% CI [0.004, 0.409]). This supports a role for intergenerational transmission, without uniquely identifying the mechanism. In this testbed, turnover improved generalization, whereas mixing partners did not demonstrate an advantage: improvement came from generations, not crowds. We report two places where our pre-registered wording overclaimed or needed tightening, and how we handled them. Plain-language summary AI systems are built by people, but many of their abilities are learned rather than programmed. We asked whether claims that an ability “emerged” can be tested in a way that could also come out negative. We wrote down five checks and built small AI agents that must invent their own code words to describe objects to each other. Before each round of experiments, we published the rules for judging its results. The agents learned languages that also worked for combinations of objects they had never seen. An apparent sudden leap in ability as the agents got bigger turned out to be mostly an effect of strict scoring: with partial credit, the improvement was gradual. Mixing up which agents talked to each other did not make them better than keeping fixed partners. But regularly replacing agents with newcomers, who had to learn the language from the others, improved results in all three rounds, and did better than the “refresh” we tested as a comparison. That supports the idea that passing a language on to new learners can improve it, although the exact mechanism is still open. The languages showed some structure, but not enough to pass our structure test. In this testbed, improvement came from generations, not crowds. Next, we plan to apply these checks to teams of language-model agents. 1. Introduction “Artificial intelligence” describes systems built by people; “emergence” concerns how capabilities arise within them. These descriptions are compatible. Learned abilities need not be explicitly programmed, but that alone does not distinguish emergence from ordinary learning and generalization. A useful account must specify what evidence supports the claim and what would count against it. Claims of abrupt “emergent abilities” of language models [8] can be artefacts of discontinuous metrics [7]. We therefore treat emergence as an empirical property with a falsifiable test. Our contributions: An operational definition with five criteria and explicit verdict rules, reported as a profile rather than a label (Section 3). EI-Bench, an open, single-GPU harness that implements the criteria for populations of communicating agents, with batched population training, turnover and perturbation controls, and bootstrap statistics (Section 4). A pre-registered, three-round study of 256 runs, with thresholds committed to git before each round (Section 5). The profile it yields is C1 PASS, C2 PASS, C3 FAIL, C4 FAIL, C5 MIRAGE. A controlled result for C2: generational turnover improves held-out performance across three rounds and outperforms the tested shrink-and-perturb control, supporting an intergenerational-transmission interpretation. 2. Related work Emergent communication. Agents trained to coordinate through a discrete channel develop conventions in the sense of Lewis signalling games [5]. Chaabouni et al. [2] showed that such languages can generalize to unseen attribute combinations without being compositional, and introduced the positional and bag-of-symbols disentanglement measures we use. Rita et al. [6] found that population effects depend on heterogeneity among agents. Iterated learning. When a language must be relearned by each new generation, transmission favours structure that is easy to learn [4]. Our turnover manipulation is a neural version of this idea. Emergent abilities and metrics. Wei et al. [8] documented abilities that appear abruptly with scale; Schaeffer et al. [7] argued many are produced by non-linear or discontinuous metrics. Our C5 criterion operationalizes that critique. Plasticity. Deep networks can lose the ability to learn under continued training [3]; re-initialization and shrink-and-perturb [1] can help restore it. This is an alternative explanation for turnover’s benefit. Our C2c comparison tests a specific plasticity-oriented control, rather than excluding all plasticity-related explanations. 3. An operational definition Let a system S be produced by a designer specification D = (architecture, objective, data, training procedure). A capability of S is emergent with respect to D to the degree that it meets the criteria in Table 1. These are proposed operational criteria, not necessary and sufficient conditions for intelligence. In this testbed, C1 is a deliberately limited proxy: generalization to held-out combinations within the same reconstruction task. Criterion Question C1 Beyond the objective Does S do something D never rewarded? C2 Ongoing emergence Does S keep developing after convergence, beyond more of the same training? C3 Interaction as source Is part of the capability caused by interaction among agents, at equal compute? C4 Self-organized structure Does unspecified structure form, above an untrained baseline? C5 Jumps survive smooth metrics If a capability seems to switch on with scale, does the jump hold under continuous metrics? Table 1. The five criteria. Each receives a verdict; together they form the system’s EI profile. Verdicts. Seeds are the unit of replication. We report means with 95% percentile bootstrap confidence intervals over seeds (10,000 resamples). A test is PASS when the whole interval clears its threshold, INCONCLUSIVE when only the mean does, and FAIL otherwise; C5 adds MIRAGE and NO-JUMP outcomes. A two-sided Mann–Whitney U test is reported as a secondary rank-based statistic. A FAIL is a verdict under the specified rule, not proof that the underlying effect is absent. In particular, an interval spanning zero does not establish equivalence between conditions. Pre-registration. Every round’s design, thresholds and verdict rules were written into DEFINITION.md and committed to a public git repository before any run of that round. Earlier verdicts are never re-scored; follow-up questions receive new, separately named tests (C2 → C2b → C2c). 4. Testbed and methods Task. Objects have A = 4 attributes with V = 8 values each (4,096 objects). A sender maps an object to a message of L = 6 symbols from a vocabulary of K = 16; a receiver must reconstruct every attribute from the message alone. Ten percent of attribute combinations (410 objects) are held out from training, while every individual value still appears in training. Nothing in D specifies the language, its structure, or what any symbol means. Agents. Senders and receivers are GRU networks (hidden size 128, embedding 32 unless swept). Symbols are discrete in the forward pass and trained with straight-through Gumbel-softmax (τ = 1). Populations use stacked parameters with a leading agent dimension, so a population of N senders and N receivers trains as one set of batched matrix multiplications. Training. Adam (learning rate 10⁻³), batch 256, 8,000 steps, with linear learning-rate decay to 0.05× over the last 20% of steps. In mixed populations, sender–receiver pairings are re-drawn every step; in the isolated control, each sender keeps one receiver. Both use identical parameters and compute. Turnover (C2). Mixed populations with N = 4 train for 24,000 steps. From step 8,000 until 85% of training, every 500 steps one randomly chosen agent (sender or receiver) is replaced by a freshly initialized agent and its Adam state is cleared (24 events per run). The learning rate decays over the final 15%. The C2 metric is the change in held-out accuracy from step 8,000 (the start of the turnover phase) to the end, compared against an arm without turnover. Controls. C2b (v0.2) adds isotropic Gaussian noise whose norm matches the displacement a reset would cause. C2c (v0.3) applies shrink-and-perturb [1]: w′ = λ·w + σ·f, with f a fresh initialization sample, σ = 0.1 and λ = ‖f‖ / ‖w‖ per parameter tensor, so the agent returns to fresh-init scale while keeping the direction of its learned weights. Metrics.