llmsays: One-Line LLM Inference with Automatic Prompt-Tier Routing and Multi-Provider Failover
llmsays is an open-source Python library that reduces large language model (LLM) inference to a single function call. It introduces a two-stage hybrid routing mechanism that classifies an input prompt into one of fourcomplexity tiers — small, medium, large, and extra_large — using a lightweight sentence-transformer model,and subsequently dispatches the query to the most appropriate provider-specific model. The library supportsfive commercial inference providers (Groq, NVIDIA NIM, OpenRouter, Fireworks AI, and Baseten) with automatic latency-aware failover, eliminating manual provider management for researchers and practitioners alike.An optional multiprocessing mode enables parallel provider queries, returning the first successful response forlatency-critical applications.
Authors
- Rudra Bhowmick
- Abhiraj Adhikary
- Anik Chand
Institutions
- Haldia Institute of Technology (IN)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-04-01
- DOI
- https://doi.org/10.5281/zenodo.19365666
- Primary Topic
- Natural Language Processing Techniques
- Type
- article
- Field-Weighted Citation Impact
- 0.00