A Preliminary Performance Analysis of LLM Inference on Edge Accelerators

Large Language Models (LLMs) have revolutionized machine learning by efficiently handling tasks like translation, text generation, summarization, and classification, making them ideal for applications such as virtual assistants and agentic workflows. However, their substantial computational requirements limit their wide adoption, particularly for smaller organizations lacking the computing infrastructure. Current LLM serving is done on the cloud, which imposes cost, latency and privacy concerns. Cloud–edge collaboration also struggles with network bottlenecks and model partitioning complexities. This work explores the potential for using edge accelerators, like Nvidia’s Jetson, for LLM inference. These compact devices have 100–1000s of GPU cores and 10+GB of shared GPU memory, allowing real-time edge inference close to the data source. We analyze and evaluate the efficiency of edge devices for LLM inference for different models and power modes, and make a case for tuning power modes to reduce power usage by ≈4–6% while matching the throughput of the highest performance mode.