Where MapReduce Catches Up: A Pre-registered Scale Ladder, the Small-File Effect Measured at 1,251.7 Percent, and a Spark Hypothesis Refuted

"Just use a script on one machine" and "you need a cluster" are both standard advice for batch document processing, usually given without numbers. This study pre-registered the numbers. On one named machine (a Dell PowerEdge R610), Hadoop MapReduce on HDFS (3.3.6, Docker pseudo-distributed cluster capped at 8 CPUs and 24 GiB) is compared with a tuned single-node baseline (same tokenizer, aggregation in DuckDB) on two document jobs over the English Wikisource corpus (648,182 documents, 1.85 GiB of extracted text), at 0.1 GiB, 1 GiB, and the full corpus. Every run's output had to hash identically to the baseline's before its time counted. Four hypotheses were frozen in advance. H1, scale crossover, is CONFIRMED: the MapReduce-to-baseline wall-time ratio for the inverted-index job falls from 2.288 at 0.1 GiB to 0.963 at the full corpus, where MapReduce is (narrowly) faster. H2, the small-file effect, is CONFIRMED at a magnitude far beyond its 15 percent pass line: at 1 GiB, storing the corpus as 1,055 files of at most 1 MiB instead of 9 files costs 1,251.7 percent of job wall time (median 7,737.209 seconds against 572.420), a factor of about 13.5. The cell had been stopped unmeasured in an earlier phase for practicality and was measured in a later phase under the owner's explicit authorization, with the deviation record public. H3, replication cost, is INCONCLUSIVE: replication factor 3 costs 29.5 percent on ingest write time but minus 0.1 percent on job time, failing the registered joint rule. H4 is REFUTED: Spark on YARN, the optional arm, is slower than MapReduce at this scale (median 1,087.0 against 954.5 seconds), not 30 percent faster. The study's claims are deliberately narrow: one machine, one corpus, one capped cluster. Within those bounds, every number was registered before it was run and every deviation is on the record.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24282234
Primary Topic
Cloud Computing and Resource Management
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Where MapReduce Catches Up: A Pre-registered Scale Ladder, the Small-File Effect Measured at 1,251.7 Percent, and a Spark Hypothesis Refuted

Jesun Ahmad Ushno
Zenodo (CERN European Organization for Nuclear Research)
Cloud Computing and Resource Management
preprint

Where MapReduce Catches Up: A Pre-registered Scale Ladder, the Small-File Effect Measured at 1,251.7 Percent, and a Spark Hypothesis Refuted

Jesun Ahmad Ushno
preprint en

Abstract

No abstract available for this paper.

Zenodo (CERN European Organization for Nuclear Research)
Cloud Computing and Resource Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon