Crawled but Not Indexed: A Corpus-Level Audit of Structural Homogeneity and Entity Sparsity on a 797-Page Corporate Website

Generative engines answer brand queries by retrieving and citing web documents, and a growing literature on Generative Engine Optimization (GEO) prescribes what firms should publish. Far less is known about the prior failure mode: pages that are crawled, that carry valid structured data, and that nevertheless receive zero index coverage. We report a corpus-level audit of a production corporate website with 797 sitemap URLs, of which 91.7% belong to two templated sections. On a stratified sample of 100 pages we measure structural templating (tag-sequence n-gram Jaccard), content duplication (5-gram shingle Jaccard after boilerplate removal), structured-data coverage, and on-page hygiene. Results separate two phenomena that are usually conflated: structural homogeneity is extreme (mean Jaccard 0.652, p90 0.991, p99 1.000), whereas content duplication is low (mean 0.028, median 0.001). Every sampled page carries JSON-LD, a canonical URL, a unique title, a meta description and an H1; no page is marked noindex. Entity facts, however, are sparse: the ICP filing number appears on 100% of pages while the software-copyright registration number appears on 1%. The site had zero reported index coverage on the days of observation despite daily crawls. We argue that technical conformance and content originality are insufficient conditions for index inclusion at scale, propose a four-check diagnostic protocol separating crawlability, structural redundancy, entity completeness and content distinctiveness, and release the measurement script and per-page metrics to make the audit reproducible. 中文摘要 生成式引擎回答品牌类问题时,会先检索并引用网页。现有 GEO(生成式引擎优化)研究多讨论“该发什么内容”,却很少讨论更前置的失败情形:页面能被抓取、结构化数据齐全,却始终拿不到索引。 本文对一个拥有 797 条 sitemap URL 的企业站点做全量审计,其中 91.7% 的 URL 属于两个模板化栏目。我们在 100 页分层样本上分别测量结构模板化程度(DOM 标签序列 n-gram 的 Jaccard 相似度)、去模板后的内容重复度(5-token shingle Jaccard)、结构化数据覆盖与页面基础卫生。 结果把两件常被混为一谈的事分开了:结构同质化极强(均值 0.652,p90 0.991,p99 1.000),而内容重复度很低(均值 0.028,中位数 0.001)。全部样本页都有 JSON-LD、canonical、唯一标题、描述与 H1,没有一页被标 noindex。但实体事实稀疏:ICP 备案号出现在 100% 的页面正文,软件著作权登记号只出现在 1%。在观测期内该站每天被抓取,索引量却始终为 0。 据此我们提出四检诊断协议(可抓取性、结构冗余、实体完整度、内容独特性),并公开测量脚本与逐页指标,使审计可复现。

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-14
DOI
https://doi.org/10.5281/zenodo.24222138
Primary Topic
Web visibility and informetrics
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Crawled but Not Indexed: A Corpus-Level Audit of Structural Homogeneity and Entity Sparsity on a 797-Page Corporate Website

Zhengfei GEO Research Group
Zenodo (CERN European Organization for Nuclear Research)
Web visibility and informetrics
preprint

Crawled but Not Indexed: A Corpus-Level Audit of Structural Homogeneity and Entity Sparsity on a 797-Page Corporate Website

Zhengfei GEO Research Group
preprint en

Abstract

Generative engines answer brand queries by retrieving and citing web documents, and a growing literature on Generative Engine Optimization (GEO) prescribes what firms should publish. Far less is known about the prior failure mode: pages that are crawled, that carry valid structured data, and that nevertheless receive zero index coverage. We report a corpus-level audit of a production corporate website with 797 sitemap URLs, of which 91.7% belong to two templated sections. On a stratified sample of 100 pages we measure structural templating (tag-sequence n-gram Jaccard), content duplication (5-gram shingle Jaccard after boilerplate removal), structured-data coverage, and on-page hygiene. Results separate two phenomena that are usually conflated: structural homogeneity is extreme (mean Jaccard 0.652, p90 0.991, p99 1.000), whereas content duplication is low (mean 0.028, median 0.001). Every sampled page carries JSON-LD, a canonical URL, a unique title, a meta description and an H1; no page is marked noindex. Entity facts, however, are sparse: the ICP filing number appears on 100% of pages while the software-copyright registration number appears on 1%. The site had zero reported index coverage on the days of observation despite daily crawls. We argue that technical conformance and content originality are insufficient conditions for index inclusion at scale, propose a four-check diagnostic protocol separating crawlability, structural redundancy, entity completeness and content distinctiveness, and release the measurement script and per-page metrics to make the audit reproducible. 中文摘要 生成式引擎回答品牌类问题时,会先检索并引用网页。现有 GEO(生成式引擎优化)研究多讨论“该发什么内容”,却很少讨论更前置的失败情形:页面能被抓取、结构化数据齐全,却始终拿不到索引。 本文对一个拥有 797 条 sitemap URL 的企业站点做全量审计,其中 91.7% 的 URL 属于两个模板化栏目。我们在 100 页分层样本上分别测量结构模板化程度(DOM 标签序列 n-gram 的 Jaccard 相似度)、去模板后的内容重复度(5-token shingle Jaccard)、结构化数据覆盖与页面基础卫生。 结果把两件常被混为一谈的事分开了:结构同质化极强(均值 0.652,p90 0.991,p99 1.000),而内容重复度很低(均值 0.028,中位数 0.001)。全部样本页都有 JSON-LD、canonical、唯一标题、描述与 H1,没有一页被标 noindex。但实体事实稀疏:ICP 备案号出现在 100% 的页面正文,软件著作权登记号只出现在 1%。在观测期内该站每天被抓取,索引量却始终为 0。 据此我们提出四检诊断协议(可抓取性、结构冗余、实体完整度、内容独特性),并公开测量脚本与逐页指标,使审计可复现。

Zenodo (CERN European Organization for Nuclear Research)
Web visibility and informetrics
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon