Meet LLMRouter: An Clever Routing System designed to Optimize LLM Inference by Dynamically Choosing essentially the most Appropriate Mannequin for Every Question

LLMRouter is an open supply routing library from the U Lab on the College of Illinois Urbana Champaign that treats mannequin choice as a firstclass system downside. It sits between purposes and a pool of LLMs and chooses a mannequin for every question primarily based on activity complexity, high quality targets, and value, all uncovered by a unified Python API and CLI. The venture ships with greater than 16 routing fashions, a knowledge technology pipeline over 11 benchmarks, and a plugin system for customized routers.

Router households and supported fashions

LLMRouter organizes routing algorithms into 4 households, Single-Spherical Routers, Multi-Spherical Routers, Personalised Routers, and Agentic Routers. Single spherical routers embrace knnrouter, svmrouter, mlprouter, mfrouter, elorouter, routerdc, automix, hybrid_llm, graphrouter, causallm_router, and the baselines smallest_llm and largest_llm. These fashions implement methods comparable to ok nearest neighbors, help vector machines, multilayer perceptrons, matrix factorization, Elo ranking, twin contrastive studying, automated mannequin mixing, and graph primarily based routing.

Multi spherical routing is uncovered by router_r1, a pre skilled occasion of Router R1 built-in into LLMRouter. Router R1 formulates multi LLM routing and aggregation as a sequential determination course of the place the router itself is an LLM that alternates between inner reasoning steps and exterior mannequin calls. It’s skilled with reinforcement studying utilizing a rule primarily based reward that balances format, end result, and value. In LLMRouter, router_r1 is out there as an additional set up goal with pinned dependencies examined on vllm==0.6.3 and torch==2.4.0.

Personalised routing is dealt with by gmtrouter, described as a graph primarily based customized router with consumer choice studying. GMTRouter represents multi flip consumer LLM interactions as a heterogeneous graph over customers, queries, responses, and fashions. It runs a message passing structure over this graph to deduce consumer particular routing preferences from few shot interplay knowledge, and experiments present accuracy and AUC good points over non customized baselines.

Agentic routers in LLMRouter prolong routing to multi step reasoning workflows. knnmultiroundrouter makes use of ok nearest neighbor reasoning over multi flip traces and is meant for complicated duties. llmmultiroundrouter exposes an LLM primarily based agentic router that performs multi step routing with out its personal coaching loop. These agentic routers share the identical configuration and knowledge codecs as the opposite router households and may be swapped by a single CLI flag.

Knowledge technology pipeline for routing datasets

LLMRouter ships with a full knowledge technology pipeline that turns customary benchmarks and LLM outputs into routing datasets. The pipeline helps 11 benchmarks, Pure QA, Trivia QA, MMLU, GPQA, MBPP, HumanEval, GSM8K, CommonsenseQA, MATH, OpenBookQA, and ARC Problem. It runs in three express phases. First, data_generation.py extracts queries and floor reality labels and creates prepare and check JSONL splits. Second, generate_llm_embeddings.py builds embeddings for candidate LLMs from metadata. Third, api_calling_evaluation.py calls LLM APIs, evaluates responses, and fuses scores with embeddings into routing data. (GitHub)

The pipeline outputs question recordsdata, LLM embedding JSON, question embedding tensors, and routing knowledge JSONL recordsdata. A routing entry contains fields comparable to task_name, question, ground_truth, metric, model_name, response, efficiency, embedding_id, and token_num. Configuration is dealt with totally by YAML, so engineers level the scripts to new datasets and candidate mannequin lists with out modifying code.

Chat interface and plugin system

For interactive use, llmrouter chat launches a Gradio primarily based chat frontend over any router and configuration. The server can bind to a customized host and port and may expose a public sharing hyperlink. Question modes management how routing sees context. current_only makes use of solely the most recent consumer message, full_context concatenates the dialogue historical past, and retrieval augments the question with the highest ok comparable historic queries. The UI visualizes mannequin decisions in actual time and is pushed by the identical router configuration used for batch inference.

LLMRouter additionally gives a plugin system for customized routers. New routers reside underneath custom_routers, subclass MetaRouter, and implement route_single and route_batch. Configuration recordsdata underneath that listing outline knowledge paths, hyperparameters, and non-compulsory default API endpoints. Plugin discovery scans the venture custom_routers folder, a ~/.llmrouter/plugins listing, and any further paths within the LLMROUTER_PLUGINS setting variable. Instance customized routers embrace randomrouter, which selects a mannequin at random, and thresholdrouter, which is a trainable router that estimates question issue.

Key Takeaways

• Routing as a firstclass abstraction: LLMRouter is an open supply routing layer from UIUC that sits between purposes and heterogeneous LLM swimming pools and centralizes mannequin choice as a price and high quality conscious prediction activity fairly than advert hoc scripts.
• 4 router households overlaying 16 plus algorithms: The library standardizes greater than 16 routers into 4 households, single spherical, multi spherical, customized, and agentic, together with knnrouter, graphrouter, routerdc, router_r1, and gmtrouter, all uncovered by a unified config and CLI.
• Multi spherical RL routing by way of Router R1: router_r1 integrates the Router R1 framework, the place an LLM router interleaves inner “assume” steps with exterior “route” calls and is skilled with a rule primarily based reward that mixes format, end result, and value to optimize efficiency price commerce offs.
• Graph primarily based personalization with GMTRouter: gmtrouter fashions customers, queries, responses and LLMs as nodes in a heterogeneous graph and makes use of message passing to be taught consumer particular routing preferences from few shot histories, reaching as much as round 21% accuracy good points and substantial AUC enhancements over robust baselines.
• Finish to finish pipeline and extensibility: LLMRouter gives a benchmark pushed knowledge pipeline, CLI for coaching and inference, a Gradio chat UI, centralized API key dealing with, and a plugin system primarily based on MetaRouter that permits groups to register customized routers whereas reusing the identical routing datasets and infrastructure.

Take a look at the GitHub Repo and Technical particulars. Additionally, be at liberty to comply with us on Twitter and don’t overlook to affix our 100k+ ML SubReddit and Subscribe to our E-newsletter. Wait! are you on telegram? now you may be a part of us on telegram as nicely.