A Hybrid Future for AI

A Hybrid Future for AIThe drive for efficiency brings large language models out of the cloud.Nvidia's rise to a $2-trillion valuation at the beginning of 2024 underlined the extraordinary computing demands of artificial intelligence systems that power ChatGPT and a host of other cloud services that create videos, music, and computer programs on demand.The power of computing and memory scaling has provided much of the impetus behind the surge in interest in generative AI based on large language models (LLMs).As models get bigger they seem to harness emergent behavior, making them more useful.But, as the growth in parameter counts has easily outstripped Moore's Law, such scaling comes at a high cost.Much of the concern around resource usage has been focused on the enormous arrays of graphics processing units (GPUs) and accelerators in training grids used to train models for weeks at a time.Inferencing has a far lower computational demand per token than training, but an influx of users can quickly overwhelm available resources.That limit appears much sooner if the queries are complex and contain thousands of tokens, each of which roughly equates to four characters of text.The problem has led to the launch of services such as GPT-For-Work and Artificial Analysis that attempt to predict the financial and energy cost of cloud-based LLMs before deployment.According to estimates from Artificial Analysis, processing a million tokens using OpenAI's GPT4-Turbo costs $15.Twenty million requests resulting in maximum-length outputs could easily translate into a bill of more than $1 million.However, the 4o version launched in the spring would likely cut that number by 50%, although it also gains efficiency by halving the maximum length of its output.The cost problem is made worse by the way in which LLMs operate during inferencing.During training, an LLM can ingest entire sentences at a time.This makes full use of the parallel arithmetic engines in GPUs.Inferencing, by contrast, demands a serial feedback loop with a single token driving each iteration."The model can't generate a new token before it has produced the last.And each token has to go through the entire network.We have to run all these weights through the compute unit each time," explained Joseph Soriaga, senior director of technology at Qualcomm AI Research.Despite the enormous memory-bandwidth requirements of LLMs, the execution units in GPUs often are woefully under-used in inferencing because they must work serially.Operators of chatbots with a large user base can gain some parallelization by batching independent queries together.Yet applications that are sensitive to latency make batching harder to justify.Also, users are increasingly concerned about privacy, knowing their questions, documents, and conversations will be uploaded to the cloud and possibly used in downstream training and fine-tuning.For all these reasons, many researchers and commercial suppliers expect more of the work of LLMs to be offloaded to users' own devices and servers.Though they cannot approach the performance of GPUs designed for cloud servers, consumer devices have the clear benefits of