Analyzing the Effects of Input Batching on the Inference Time of Vision Transformer Models

Data batching is a fundamental technique in machine learning and deep learning that enhances model execution by processing multiple data samples simultaneously. This approach optimizes memory usage, improves computational efficiency, and accelerates both training and inference by leveraging modern hardware capabilities like GPUs and TPUs. Batching enables vectorized computations, reducing processing overhead and maximizing hardware utilization. It also stabilizes training by averaging gradient updates, leading to smoother optimization and better convergence, especially in large-scale neural networks. Additionally, batching improves model generalization by exposing it to diverse examples and enables techniques like Batch Normalization, further enhancing stability and performance. This paper investigates Vision Transformer (ViT) models with varying batch sizes, demonstrating that increasing the batch size significantly reduces inference time. Notably, a batch size of 16 achieves an 83% reduction in inference time compared to single-sample processing, underscoring the substantial efficiency gains from batching.