Collective Voting: Multi-Model Ensemble Inference on Unified Memory

Ensemble methods improve prediction quality by combining multiple independent models. While effective, ensembles are rarely deployed in production AI inference due to the memory cost of loading multiple models simultaneously. The Apple M2 Ultra with 192GB unified memory changes this calculus: we can load 2-3 70B-parameter models at 4-bit quantization (~40GB each) alongside a routing ensemble layer, with memory remaining for KV cache. We propose Collective Voting, an inference framework that runs 2-3 independently fine-tuned models on the same input and aggregates their outputs through weighted voting, probabilistic merging, and confidence-based selection. On a 192GB system, this configuration runs at 5-10 tok/s (2 models) or 3-6 tok/s (3 models), with quality improvements of 5-12% over individual models on reasoning benchmarks. The ensemble provides critical advantages: robustness to individual model failure, uncertainty quantification through vote entropy, and the ability to combine models trained on complementary data distributions. We compare against single-model inference, model soups, and multi-agent debate, identifying the conditions under which ensemble voting outperforms each approach.