LayerForge: Deterministic Layer Decomposition for AI Output Convergence — An Empirical Report on Newman Modularity Outperforming CPM in Small-N Text Domains

Conversational AI output tends to include content beyond what the questioner is actually concerned with. We report on LayerForge, a deterministic pipeline that decomposes a corpus into 4±1 thematic layers and returns only the layer(s) relevant to a query. The pipeline combines five established components — Cowan's working-memory upper bound, Newman modularity for community detection, SCA-style per-layer distillation, HERCULES-style hierarchical KMeans, and a deterministic core with no LLM in the analysis loop — chosen so that "AI noise filtering" does not itself depend on an AI. We re-implemented a CPM (Constant Potts Model) backend (MIT-pure, self-implemented) to test whether the theoretically resolution-limit-free alternative would outperform Newman in this small-N text domain (operational target N=20-40, external scaling check up to N=100 on 20 Newsgroups). Across 119-row dual-method correlation sweep, 196-cell N×K heatmap, 95-row ARI/NMI comparison, and the external 20-Newsgroups benchmark, the Newman backend exceeded CPM by a large effect size in our test range (H_struct exact match 87.5% vs 19%; 20NG ARI 0.430 vs 0.239 at default K=3, Newman reaches 0.557 when K=4 is forced; same direction at N=100), against the naïve theoretical expectation that CPM should win. Positioning: LayerForge shares its core substrate (sentence-embedding similarity + community detection) with prior work in clustering-based topic modeling, notably G2T (Zhang et al. 2023). The contribution of this paper is twofold: (i) the implementation report of combining SCA decomposition, HERCULES hierarchical skeleton, the 4±1 cognitive constraint, and a Mode A/B/C operational layer, and (ii) the systematic empirical comparison of Newman vs CPM in the small-N text regime, including a three-failure-mode mechanism analysis. Concurrently with this work, Bohdal et al. (Samsung 2026) independently arrived at the same problem space on the LLM memory token side; we treat this convergence as evidence that the design space is being identified by multiple independent groups. We report an empirically derived method-selection rule (concept-only; runtime implementation deferred because in the studied domain the rule would output "newman" in essentially all cells). The artifact is published as a Claude Code skill (LayerForge itself does not hold an API key; LLM access at Boundary 1/2 is mediated through Claude Code), and is reproducible end-to-end.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-05-13
DOI
https://doi.org/10.5281/zenodo.20155668
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

LayerForge: Deterministic Layer Decomposition for AI Output Convergence — An Empirical Report on Newman Modularity Outperforming CPM in Small-N Text Domains

Yasuhiro Kuroki
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

LayerForge: Deterministic Layer Decomposition for AI Output Convergence — An Empirical Report on Newman Modularity Outperforming CPM in Small-N Text Domains

Yasuhiro Kuroki
preprint en

Abstract

Conversational AI output tends to include content beyond what the questioner is actually concerned with. We report on LayerForge, a deterministic pipeline that decomposes a corpus into 4±1 thematic layers and returns only the layer(s) relevant to a query. The pipeline combines five established components — Cowan's working-memory upper bound, Newman modularity for community detection, SCA-style per-layer distillation, HERCULES-style hierarchical KMeans, and a deterministic core with no LLM in the analysis loop — chosen so that "AI noise filtering" does not itself depend on an AI. We re-implemented a CPM (Constant Potts Model) backend (MIT-pure, self-implemented) to test whether the theoretically resolution-limit-free alternative would outperform Newman in this small-N text domain (operational target N=20-40, external scaling check up to N=100 on 20 Newsgroups). Across 119-row dual-method correlation sweep, 196-cell N×K heatmap, 95-row ARI/NMI comparison, and the external 20-Newsgroups benchmark, the Newman backend exceeded CPM by a large effect size in our test range (H_struct exact match 87.5% vs 19%; 20NG ARI 0.430 vs 0.239 at default K=3, Newman reaches 0.557 when K=4 is forced; same direction at N=100), against the naïve theoretical expectation that CPM should win. Positioning: LayerForge shares its core substrate (sentence-embedding similarity + community detection) with prior work in clustering-based topic modeling, notably G2T (Zhang et al. 2023). The contribution of this paper is twofold: (i) the implementation report of combining SCA decomposition, HERCULES hierarchical skeleton, the 4±1 cognitive constraint, and a Mode A/B/C operational layer, and (ii) the systematic empirical comparison of Newman vs CPM in the small-N text regime, including a three-failure-mode mechanism analysis. Concurrently with this work, Bohdal et al. (Samsung 2026) independently arrived at the same problem space on the LLM memory token side; we treat this convergence as evidence that the design space is being identified by multiple independent groups. We report an empirically derived method-selection rule (concept-only; runtime implementation deferred because in the studied domain the rule would output "newman" in essentially all cells). The artifact is published as a Claude Code skill (LayerForge itself does not hold an API key; LLM access at Boundary 1/2 is mediated through Claude Code), and is reproducible end-to-end.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.