The Emulator Is a Claim: A Pre-registered Audit of 30 Documented boto3 Behaviours Against moto 5.2.5, with 24 Exact Matches and Six Named Departures

Small teams test cloud code against local emulators because cloud bills are real, but an emulator is a claim about a service, made by people who did not build the service, and the claim is uneven. This study measured exactly where one widely used emulator's claim holds. Under a pre-registration locked before any run existed (the root commit of the study repository), 30 documented behaviours of the S3, SQS, and Lambda patterns a working engineer actually uses were frozen, each with its expected result written from AWS's own documentation, and scored against a local moto 5.2.5 server driven by boto3 1.43.126: MATCHES (every assertion holds), DIVERGES (the operation runs but an assertion differs), or UNSUPPORTED (the emulator refuses or cannot execute it, with its error quoted). H1 is CONFIRMED: 24 of 30 rows score MATCHES (80.0 percent), above the frozen 70 percent line. H2 is REFUTED: the hypothesis that every service has at least one diverging or unsupported row fails on SQS, whose entire frozen surface, all 10 rows including visibility timeout, dead-letter redrive, FIFO deduplication and ordering, and the documented message-attribute MD5, matched the documents exactly. H3 is CONFIRMED: the two queue semantics production code relies on most, visibility timeout (Q-02) and dead-letter redrive (Q-04), both score MATCHES. The six departures are the primary deliverable. Three divergences: presigned URLs signed for a custom endpoint carry Signature Version 2 style parameters (AWSAccessKeyId, Signature, Expires), not the documented SigV4 X-Amz parameters, though the URL fetches the exact object bytes; a put with a deliberately wrong Content-MD5 is accepted and stored, where real S3 rejects it with BadDigest, so an upload-integrity test that passes against moto proves nothing about that check; and CreateFunction accepts the invalid runtime string "python2.7-removed-runtime", where real Lambda rejects it with InvalidParameterValueException. Three unsupported rows: synchronous Lambda invocation and the failing-handler invocation both return HTTP 200 with the emulator's own "error running docker" text as payload, because moto executes zip functions through Docker and the machine of record has none, so the behaviour engineers test most is exactly the behaviour this setup cannot evidence; and S3 Select answers the documented CAST query with HTTP 500, because moto's SQL parser raises ValueError on the CAST where-clause. Nothing in the study touches real AWS: every client was forced to a loopback endpoint by a guarded factory, and the paper claims evidence about the emulator, transferable to the service only through the documents both were scored against.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24328286
Primary Topic
Cloud Computing and Resource Management
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

The Emulator Is a Claim: A Pre-registered Audit of 30 Documented boto3 Behaviours Against moto 5.2.5, with 24 Exact Matches and Six Named Departures

Jesun Ahmad Ushno
Zenodo (CERN European Organization for Nuclear Research)
Cloud Computing and Resource Management
preprint

The Emulator Is a Claim: A Pre-registered Audit of 30 Documented boto3 Behaviours Against moto 5.2.5, with 24 Exact Matches and Six Named Departures

Jesun Ahmad Ushno
preprint en

Abstract

Small teams test cloud code against local emulators because cloud bills are real, but an emulator is a claim about a service, made by people who did not build the service, and the claim is uneven. This study measured exactly where one widely used emulator's claim holds. Under a pre-registration locked before any run existed (the root commit of the study repository), 30 documented behaviours of the S3, SQS, and Lambda patterns a working engineer actually uses were frozen, each with its expected result written from AWS's own documentation, and scored against a local moto 5.2.5 server driven by boto3 1.43.126: MATCHES (every assertion holds), DIVERGES (the operation runs but an assertion differs), or UNSUPPORTED (the emulator refuses or cannot execute it, with its error quoted). H1 is CONFIRMED: 24 of 30 rows score MATCHES (80.0 percent), above the frozen 70 percent line. H2 is REFUTED: the hypothesis that every service has at least one diverging or unsupported row fails on SQS, whose entire frozen surface, all 10 rows including visibility timeout, dead-letter redrive, FIFO deduplication and ordering, and the documented message-attribute MD5, matched the documents exactly. H3 is CONFIRMED: the two queue semantics production code relies on most, visibility timeout (Q-02) and dead-letter redrive (Q-04), both score MATCHES. The six departures are the primary deliverable. Three divergences: presigned URLs signed for a custom endpoint carry Signature Version 2 style parameters (AWSAccessKeyId, Signature, Expires), not the documented SigV4 X-Amz parameters, though the URL fetches the exact object bytes; a put with a deliberately wrong Content-MD5 is accepted and stored, where real S3 rejects it with BadDigest, so an upload-integrity test that passes against moto proves nothing about that check; and CreateFunction accepts the invalid runtime string "python2.7-removed-runtime", where real Lambda rejects it with InvalidParameterValueException. Three unsupported rows: synchronous Lambda invocation and the failing-handler invocation both return HTTP 200 with the emulator's own "error running docker" text as payload, because moto executes zip functions through Docker and the machine of record has none, so the behaviour engineers test most is exactly the behaviour this setup cannot evidence; and S3 Select answers the documented CAST query with HTTP 500, because moto's SQL parser raises ValueError on the CAST where-clause. Nothing in the study touches real AWS: every client was forced to a loopback endpoint by a guarded factory, and the paper claims evidence about the emulator, transferable to the service only through the documents both were scored against.

Zenodo (CERN European Organization for Nuclear Research)
Cloud Computing and Resource Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon