What a Backtest Costs: Throughput, Memory, Scaling and Reproducibility of Four Backtesting Engines on a 64-Core CPU

We measure the wall time and memory of one backtest in Backtrader, NautilusTrader, QuantConnect LEAN and Reamer Research. We also measure how both grow with bar count (50,000 and 500,000 bars), ticker breadth (10 to 100 tickers) and parallel copies (1 to 64). Every engine runs the same two strategies on the same data, on one AMD EPYC 9575F. Versions are pinned and the raw results are public. The author builds and sells Reamer Research. Before seeing any result, we stated four outcomes that would count against it. Reamer Research was fastest in every configuration. On 500,000 bars of the realistic strategy, its C++ path processed 1.39 million bars per second and its Python path 148,000, against 6,500 to 37,000 for the other engines. At 100 tickers its Python path finished in 14.9 s, against 94.8 s for the next fastest engine. With 64 copies in parallel, the C++ path completed 332,000 backtests per hour, about 30 times the next engine's 11,200. The C++ path also used the least memory of any engine in every configuration it ran. Every engine reproduced its results exactly across repetitions. No Reamer Research run failed, while LEAN hung in 14 of 22 multi-ticker attempts. Two of the four adverse outcomes occurred. Above about 270,000 bars of the minimal strategy, Backtrader used less memory than Reamer Research's Python path: 209 MB against 309 MB at 500,000 bars. The C++ path had the lowest parallel efficiency at 64 copies, 0.53 against 0.63–0.72, although its throughput kept rising. We also report a quadratic cost in NautilusTrader 1.231.0's PnL recording, a multi-instrument cost in its position snapshots, and a start-up hang of LEAN under concurrency. Raw results, runners, dataset builders and the table script: https://doi.org/10.6084/m9.figshare.34785729