An intelligent gateway that analyses every incoming prompt and automatically routes it to the right model — a fast local LLM for simple tasks, a cloud API for complex ones.

Built for developers who run powerful hardware locally (tested on a MacBook with 48 GB RAM) and want to reduce cloud API usage without sacrificing quality on tasks that actually need it.

Your prompt
    │
    ▼
┌─────────────────────┐
│  Complexity Scorer  │  heuristic pipeline, <1 ms
│       0 – 100       │
└─────────────────────┘
    │               │
 simple          complex
    │               │
    ▼               ▼
 Local LLM      Cloud API
(LM Studio)  (Claude / Gemini)

• Auto-routing — scores every prompt 0–100 on a heuristic complexity scale and sends it to the right backend
• OpenAI-compatible proxy — run as a server on :8080/v1 and point any tool that talks to OpenAI directly at it, no code changes needed
• Interactive CLI chat — python chat.py for instant streaming conversations
• Supports Claude and Gemini as cloud backends
• Automatic fallback — if LM Studio is offline, requests silently fall back to the cloud
• Force routing — override per-request via X-Force-Route: local / cloud header or CLI flags
• Transparent logging — every request prints its score, target, and reason

git clone <your-repo-url>
cd llm-router

python3 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -r requirements.txt

cp .env.example .env

Open .env and fill in your values:

# Local model — match this to whatever is loaded in LM Studio
LOCAL_MODEL=qwen2.5-32b-instruct

# Choose your cloud backend: "anthropic" or "gemini"
CLOUD_PROVIDER=gemini
GEMINI_API_KEY=AIza...
# or
CLOUD_PROVIDER=anthropic
ANTHROPIC_API_KEY=sk-ant-...

Open LM Studio → load a model → go to Local Server tab → click Start Server.
The server starts on http://localhost:1234 by default.

python chat.py              # auto-route based on complexity
python chat.py --local      # force everything to the local model
python chat.py --cloud      # force everything to the cloud

In-chat commands:

Run the router as a local OpenAI-compatible API server:

python server.py

Then point any OpenAI SDK client at http://localhost:8080/v1:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="router")

response = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarize this email: ..."}]
)
print(response.choices[0].message.content)

# Via header
response = client.chat.completions.create(
    model="auto",
    messages=[...],
    extra_headers={"X-Force-Route": "local"},  # or "cloud"
)

Every prompt is scored 0–100 using a fast heuristic pipeline (no model call needed):

Scores at or above COMPLEXITY_THRESHOLD (default: 50) go to the cloud.

# Send more to local (trust your local model more)
COMPLEXITY_THRESHOLD=65

# Send more to cloud (higher bar for local)
COMPLEXITY_THRESHOLD=35

# Hard cap: prompts longer than this always go to cloud
MAX_LOCAL_TOKENS=3000

llm-router/
├── chat.py              # Interactive CLI chat
├── server.py            # FastAPI proxy server
├── config.py            # Centralised configuration
├── router/
│   ├── classifier.py    # Heuristic complexity scorer
│   ├── router.py        # Routing logic + availability cache
│   ├── local_client.py  # LM Studio adapter (OpenAI SDK)
│   └── cloud_client.py  # Claude + Gemini adapters
├── test_router.py       # Classifier test suite
├── requirements.txt
├── .env.example         # Copy to .env and fill in keys
└── .gitignore

python test_router.py           # classifier accuracy only
python test_router.py --live    # classifier + real model calls