Program Scalability Analysis for LLM Endpoints: Ahmdal's Law Analysis of Parallelizability Benefits of LLM Completions Endpoints*

This paper presents an analysis of program scalability when using LLM endpoints for parallel chunked completions. By applying Amdahl's Law, we benchmark the performance and scalability of different parallel execution strategies. Additionally, we show that program instrumentation helps Gustafson's scaled speedup formulation to quantify the elusive quality in Amdahl's Law. Furthermore, we report that without combining multiple completions into an ensemble, the results are not trustworthy. We demonstrate a methodology that can help achieve more stable completions by repeated execution with ensemble and identify the optimal degree of parallelism. The study aims to provide insights into optimizing the use of LLM endpoints for applications with large input limits.