Tiny Large Language Models in Embedded Nvidia Portable Hardware Comparative Analysis

This study leverages the NVIDIA platform, a hardware with constrained computational resources, to evaluate the real-time performance of various small language models as local assistants, functioning without Internet access. The research employed a novel four-phase methodology, beginning with the selection of small language models for evaluation, followed by the design of a reproducible test protocol for future studies. This protocol incorporates both quantitative and qualitative assessment criteria, including latency, power consumption, memory usage, accuracy, creativity, narrative coherence, structural adherence, and Spanish performance. In the third phase, the selected models were locally embedded and executed on the hardware to compare their respective performances. Finally, the suitability of each model for real-time applications was analyzed, leading to the development of a comprehensive test protocol. The findings indicate that, of the nine models evaluated, Qwen2.5 with 3 billion parameters emerges as the optimal choice when resources allow, while Qwen2.5 with 0.5 billion parameters provides a viable alternative for scenarios with severe resource limitations.