Assessing the Impracticality of Training and Finetuning Large Language Models on Consumer GPUs: Limitations in Time, Scalability, and Efficiency

High-performance GPUs like the NVIDIA A100 and H100 offer significant FLOP rates, accelerating complex model training, yet they remain out of reach for a broader audience. Conversely, consumer GPUs(AMD Radeon W7900, NVIDIA GeForce GTX 1080), while not as optimized for deep learning tasks, provide accessible and cost-effective alternatives for these workloads. By benchmarking the performance of different attention mechanisms on different consumer GPUS and provides statistics to assess the feasibility and practicality of the training LLMs on the same. This paper underscores the potential for consumer GPUs to support certain workloads but emphasizes the need to make LLM training more accessible. Insights on optimizing AI hardware for scalable, cost-effective generative AI applications are discussed.