llmsays: One-Line LLM Inference with Automatic Prompt-Tier Routing and Multi-Provider Failover

llmsays is an open-source Python library that reduces large language model (LLM) inference to a single function call. It introduces a two-stage hybrid routing mechanism that classifies an input prompt into one of fourcomplexity tiers — small, medium, large, and extra_large — using a lightweight sentence-transformer model,and subsequently dispatches the query to the most appropriate provider-specific model. The library supportsfive commercial inference providers (Groq, NVIDIA NIM, OpenRouter, Fireworks AI, and Baseten) with automatic latency-aware failover, eliminating manual provider management for researchers and practitioners alike.An optional multiprocessing mode enables parallel provider queries, returning the first successful response forlatency-critical applications.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-04-01
DOI
https://doi.org/10.5281/zenodo.19365666
Primary Topic
Natural Language Processing Techniques
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

llmsays: One-Line LLM Inference with Automatic Prompt-Tier Routing and Multi-Provider Failover

Rudra Bhowmick, Abhiraj Adhikary, Anik Chand
Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
article

llmsays: One-Line LLM Inference with Automatic Prompt-Tier Routing and Multi-Provider Failover

Rudra Bhowmick, Abhiraj Adhikary, Anik Chand
article en

Abstract

llmsays is an open-source Python library that reduces large language model (LLM) inference to a single function call. It introduces a two-stage hybrid routing mechanism that classifies an input prompt into one of fourcomplexity tiers — small, medium, large, and extra_large — using a lightweight sentence-transformer model,and subsequently dispatches the query to the most appropriate provider-specific model. The library supportsfive commercial inference providers (Groq, NVIDIA NIM, OpenRouter, Fireworks AI, and Baseten) with automatic latency-aware failover, eliminating manual provider management for researchers and practitioners alike.An optional multiprocessing mode enables parallel provider queries, returning the first successful response forlatency-critical applications.

Zenodo (CERN European Organization for Nuclear Research)
Haldia Institute of Technology (IN)
Quality Education
Openalex Percentile: Top 27%
Natural Language Processing Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.