Statistical E-Commerce Search Evaluation with LLM Relevance Judge Panels and CI Quality Gates: A Technical Analysis of llm-search-eval v0.5.0

OverviewThis technical report presents the architecture, methodology, statistical evaluation procedures, and empirical benchmark results of llm-search-eval v0.5.0, an open-source Python library for evaluating e-commerce product search systems.The framework combines classical Information Retrieval (IR) metrics with multi-judge Large Language Model (LLM) relevance assessments. It is designed to address practical challenges in offline search evaluation, including sparse human relevance judgments, label depletion, evaluator variability, statistical uncertainty, and evaluation workflow integration.System ArchitectureThe report describes a four-stage label resolution hierarchy:Human Gold Labels: Human-provided query relevance judgments take precedence over automated assessments.Content-Addressed Judgment Cache: Previously computed judgments are reused through content-addressed caching.LLM Judge Panels: Unlabeled and uncached query-product pairs can be evaluated by configured LLM judges.Fallback and Coverage Tracking: Unresolved items receive a computational fallback score while coverage gaps are explicitly recorded. This fallback does not represent an explicit judgment of irrelevance.The report also documents model-version fingerprinting, evaluator drift monitoring, inter-rater agreement, and pairwise preference evaluation.Evaluation MethodologyThe framework implements standard rank-aware Information Retrieval metrics, including DCG, nDCG, Precision, Recall, Mean Reciprocal Rank (MRR), and Mean Average Precision (MAP).Statistical procedures include paired sign-flip permutation testing and query-level bootstrap confidence intervals. The report documents a two-sided permutation test using 2,000 random permutations with seed 13, and 95% percentile bootstrap confidence intervals based on 1,000 query-level resamples with replacement.Empirical BenchmarkThe report presents a bilingual electronics product-search benchmark comprising:10 queries: 5 English and 5 Persian22 products30 human gold relevance labels49 unique judged query-product pairs19 additional pairs assessed using a deterministic three-judge mock panelThree retrieval approaches are compared: BM25, Semantic retrieval, and Hybrid retrieval.The reported nDCG@10 scores are:BM25: 0.5305Semantic: 0.9865Hybrid: 0.9775The report also presents confidence intervals, paired comparisons against the BM25 baseline, and language-specific benchmark results.LimitationsThe benchmark is limited in scale and uses synthetic mock judgments for filling evaluation gaps. It does not establish general performance superiority across broader e-commerce search workloads. The report also identifies benchmark author bias, the absence of online behavioral validation, and the lack of latency and memory benchmarks as limitations.Software and ReproducibilityThe report documents llm-search-eval v0.5.0 and provides links to its source repository, documentation, package distribution, benchmark dataset, and software archive.Software DOI: 10.5281/zenodo.23944666Report DOI: 10.6084/m9.figshare.35004088Peer-review status: Technical report, not peer-reviewed.

Authors

Publication Details

Journal
Figshare
Published
2026-11-13
DOI
https://doi.org/10.6084/m9.figshare.35004088
Primary Topic
Information Retrieval and Search Behavior
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Statistical E-Commerce Search Evaluation with LLM Relevance Judge Panels and CI Quality Gates: A Technical Analysis of llm-search-eval v0.5.0

Mohammad Naser Sadrmanesh
3 links
Figshare
Information Retrieval and Search Behavior
preprint

Statistical E-Commerce Search Evaluation with LLM Relevance Judge Panels and CI Quality Gates: A Technical Analysis of llm-search-eval v0.5.0

Mohammad Naser Sadrmanesh
preprint en

Abstract

OverviewThis technical report presents the architecture, methodology, statistical evaluation procedures, and empirical benchmark results of llm-search-eval v0.5.0, an open-source Python library for evaluating e-commerce product search systems.The framework combines classical Information Retrieval (IR) metrics with multi-judge Large Language Model (LLM) relevance assessments. It is designed to address practical challenges in offline search evaluation, including sparse human relevance judgments, label depletion, evaluator variability, statistical uncertainty, and evaluation workflow integration.System ArchitectureThe report describes a four-stage label resolution hierarchy:Human Gold Labels: Human-provided query relevance judgments take precedence over automated assessments.Content-Addressed Judgment Cache: Previously computed judgments are reused through content-addressed caching.LLM Judge Panels: Unlabeled and uncached query-product pairs can be evaluated by configured LLM judges.Fallback and Coverage Tracking: Unresolved items receive a computational fallback score while coverage gaps are explicitly recorded. This fallback does not represent an explicit judgment of irrelevance.The report also documents model-version fingerprinting, evaluator drift monitoring, inter-rater agreement, and pairwise preference evaluation.Evaluation MethodologyThe framework implements standard rank-aware Information Retrieval metrics, including DCG, nDCG, Precision, Recall, Mean Reciprocal Rank (MRR), and Mean Average Precision (MAP).Statistical procedures include paired sign-flip permutation testing and query-level bootstrap confidence intervals. The report documents a two-sided permutation test using 2,000 random permutations with seed 13, and 95% percentile bootstrap confidence intervals based on 1,000 query-level resamples with replacement.Empirical BenchmarkThe report presents a bilingual electronics product-search benchmark comprising:10 queries: 5 English and 5 Persian22 products30 human gold relevance labels49 unique judged query-product pairs19 additional pairs assessed using a deterministic three-judge mock panelThree retrieval approaches are compared: BM25, Semantic retrieval, and Hybrid retrieval.The reported nDCG@10 scores are:BM25: 0.5305Semantic: 0.9865Hybrid: 0.9775The report also presents confidence intervals, paired comparisons against the BM25 baseline, and language-specific benchmark results.LimitationsThe benchmark is limited in scale and uses synthetic mock judgments for filling evaluation gaps. It does not establish general performance superiority across broader e-commerce search workloads. The report also identifies benchmark author bias, the absence of online behavioral validation, and the lack of latency and memory benchmarks as limitations.Software and ReproducibilityThe report documents llm-search-eval v0.5.0 and provides links to its source repository, documentation, package distribution, benchmark dataset, and software archive.Software DOI: 10.5281/zenodo.23944666Report DOI: 10.6084/m9.figshare.35004088Peer-review status: Technical report, not peer-reviewed.

Figshare
Information Retrieval and Search Behavior
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon