The Emulator Is a Claim: A Pre-registered Audit of 30 Documented boto3 Behaviours Against moto 5.2.5, with 24 Exact Matches and Six Named Departures
Small teams test cloud code against local emulators because cloud bills are real, but an emulator is a claim about a service, made by people who did not build the service, and the claim is uneven. This study measured exactly where one widely used emulator's claim holds. Under a pre-registration locked before any run existed (the root commit of the study repository), 30 documented behaviours of the S3, SQS, and Lambda patterns a working engineer actually uses were frozen, each with its expected result written from AWS's own documentation, and scored against a local moto 5.2.5 server driven by boto3 1.43.126: MATCHES (every assertion holds), DIVERGES (the operation runs but an assertion differs), or UNSUPPORTED (the emulator refuses or cannot execute it, with its error quoted). H1 is CONFIRMED: 24 of 30 rows score MATCHES (80.0 percent), above the frozen 70 percent line. H2 is REFUTED: the hypothesis that every service has at least one diverging or unsupported row fails on SQS, whose entire frozen surface, all 10 rows including visibility timeout, dead-letter redrive, FIFO deduplication and ordering, and the documented message-attribute MD5, matched the documents exactly. H3 is CONFIRMED: the two queue semantics production code relies on most, visibility timeout (Q-02) and dead-letter redrive (Q-04), both score MATCHES. The six departures are the primary deliverable. Three divergences: presigned URLs signed for a custom endpoint carry Signature Version 2 style parameters (AWSAccessKeyId, Signature, Expires), not the documented SigV4 X-Amz parameters, though the URL fetches the exact object bytes; a put with a deliberately wrong Content-MD5 is accepted and stored, where real S3 rejects it with BadDigest, so an upload-integrity test that passes against moto proves nothing about that check; and CreateFunction accepts the invalid runtime string "python2.7-removed-runtime", where real Lambda rejects it with InvalidParameterValueException. Three unsupported rows: synchronous Lambda invocation and the failing-handler invocation both return HTTP 200 with the emulator's own "error running docker" text as payload, because moto executes zip functions through Docker and the machine of record has none, so the behaviour engineers test most is exactly the behaviour this setup cannot evidence; and S3 Select answers the documented CAST query with HTTP 500, because moto's SQL parser raises ValueError on the CAST where-clause. Nothing in the study touches real AWS: every client was forced to a loopback endpoint by a guarded factory, and the paper claims evidence about the emulator, transferable to the service only through the documents both were scored against.
Authors
- Jesun Ahmad Ushno (ORCID: https://orcid.org/0000-0002-6070-9025)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.5281/zenodo.24328286
- Primary Topic
- Cloud Computing and Resource Management
- Type
- preprint