AGENTS.md
Comprehensive module mapping, state machines, and architectural patterns for vLLM Studio.
Repo Conventions (Strict)
- All files must be 60 LOC or less unless explicitly critical; prefer splitting into subdirectories.
- Any file that exceeds 60 LOC must start with a CRITICAL marker (e.g.,
// CRITICAL,# CRITICAL, or<!-- CRITICAL -->). - Files that cannot accept comment markers (e.g., lockfiles, LICENSE,
.gitignore,.env.example) are implicitly critical. - For shebang scripts, place the CRITICAL marker on the second line.
- Never use camelCase or PascalCase for file or directory names; use kebab-case only.
- Keep ≤20 files per directory; create subdirectories when needed.
Module Dependency Graph
┌─────────────┐
│ cli.py │
└──────┬──────┘
│
┌──────▼──────┐
│ app.py │ ◄── Entry point, lifespan
└──────┬──────┘
│
┌────────────────────────────┼────────────────────────────┐
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ config.py │ │ events.py │ │ store.py │
│ Settings │ │ EventManager │ │ *Store │
└───────────────┘ └───────────────┘ └───────────────┘
│ │ │
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ models.py │ │ routes/* │◄──────────│ models.py │
│ Recipe, MCP │◄──────────│ API handlers │ │ Pydantic │
└───────────────┘ └───────┬───────┘ └───────────────┘
│ │
│ │
▼ ▼
┌───────────────┐ ┌───────────────┐
│ backends.py │◄──────────│ process.py │
│ Cmd builders │ │ Launch/Evict │
└───────────────┘ └───────────────┘
│
▼
┌───────────────┐
│ gpu.py │
│ GPU detection │
└───────────────┘
State Machines
1. Model Launch State Machine
┌─────────────────┐
│ IDLE │
└────────┬────────┘
│ POST /launch/{recipe_id}
▼
┌─────────────────┐
┌──────│ CHECK_LOCK │──────┐
│ └─────────────────┘ │
│ locked │ available
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ WAIT_LOCK (2s) │────────────▶│ ACQUIRE_LOCK │
└────────┬────────┘ timeout └────────┬────────┘
│ preempt │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ CANCEL_OTHER │ │ EVICTING │ ◄── Progress: 0%
└────────┬────────┘ └────────┬────────┘
│ │ kill process, wait 1s
└──────────────┬────────────────┘
▼
┌─────────────────┐
│ CHECK_CANCEL │──────────────┐
└────────┬────────┘ │ cancelled
│ continue ▼
▼ ┌─────────────┐
┌─────────────────┐ │ CANCELLED │
│ LAUNCHING │ ◄── └─────────────┘
│ Progress: 25% │
└────────┬────────┘
│ subprocess.Popen, wait 3s
▼
┌─────────────────┐
│ CHECK_STABILITY │
└────────┬────────┘
│
┌──────────────┴──────────────┐
│ crashed │ stable
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ READ_LOGS │ │ WAITING │ ◄── Progress: 50%
└────────┬────────┘ └────────┬────────┘
│ │ poll /health (300s timeout)
▼ │
┌─────────────────┐ ┌────────┴────────┐
│ ERROR │ │ │
└─────────────────┘ ▼ ▼
┌─────────────┐ ┌─────────────────┐
│ READY │ │ TIMEOUT │
│ Progress: │ └─────────────────┘
│ 100% │
└─────────────┘
Events emitted:
launch_progressat each state transition (0%, 25%, 50%, 100%)statuswhen model becomes readyerroron failure
2. Chat Message Flow
┌─────────────────────────────────────────────────────────────────┐
│ FRONTEND (React) │
├─────────────────────────────────────────────────────────────────┤
│ │
│ User Input ──▶ buildAPIMessages() ──▶ getOpenAITools() │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ POST /api/chat │ │
│ │ { messages, model, tools } │ │
│ └───────────────────────┬─────────────────────────┘ │
│ │ │
└──────────────────────────┼──────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ NEXT.JS API ROUTE │
├─────────────────────────────────────────────────────────────────┤
│ /api/chat/route.ts │
│ │ │
│ ▼ │
│ Stream to LiteLLM:4100/v1/chat/completions │
└───────────────────────┬─────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ LITELLM (4100) │
├─────────────────────────────────────────────────────────────────┤
│ config/litellm.yaml routing │
│ │ │
│ ├──▶ GLM models ──▶ vLLM with glm45 parser │
│ ├──▶ MiniMax models ──▶ vLLM with minimax parser │
│ ├──▶ INTELLECT-3 ──▶ vLLM with hermes parser │
│ └──▶ * (wildcard) ──▶ vLLM default │
│ │
│ Callbacks: │
│ - tool_call_handler (MCP processing) │
│ - prometheus_callback (metrics) │
│ - spend_logs (PostgreSQL) │
└───────────────────────┬─────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ vLLM/SGLang (8000) │
├─────────────────────────────────────────────────────────────────┤
│ Inference with: │
│ - Tool call parsing (function calling) │
│ - Reasoning token extraction │
│ - Streaming response │
└───────────────────────┬─────────────────────────────────────────┘
│
▼ SSE Stream
┌─────────────────────────────────────────────────────────────────┐
│ FRONTEND │
├─────────────────────────────────────────────────────────────────┤
│ parseSSEEvents() processes: │
│ │ │
│ ├──▶ type: "text" ──▶ Append to message content │
│ ├──▶ type: "tool_calls" ──▶ Display in ToolBelt │
│ ├──▶ type: "thinking" ──▶ Show in thinking modal │
│ └──▶ type: "error" ──▶ Display error │
│ │
│ If tool_calls present: │
│ └──▶ executeMCPTool() ──▶ Loop back with tool results │
│ │
│ Save to ChatStore via POST /chats/{id}/messages │
└─────────────────────────────────────────────────────────────────┘
3. MCP Tool Execution Flow
┌─────────────────┐
│ Model Request │
│ tool_call: │
│ { │
│ name: "exa__ │
│ web_search" │
│ arguments:{} │
│ } │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Parse Tool Name │
│ server = "exa" │
│ tool = "web_ │
│ search" │
└────────┬────────┘
│
▼
┌─────────────────┐ POST /mcp/tools/{server}/{tool}
│ API Request │────────────────────────────────────▶
└─────────────────┘
│
┌───────────────────────────────┘
│
▼
┌─────────────────┐
│ Controller │
│ routes/mcp.py │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Load MCPServer │
│ from MCPStore │
└────────┬────────┘
│
▼
┌─────────────────────────────────────────┐
│ _run_mcp_command() │
├─────────────────────────────────────────┤
│ │
│ 1. spawn subprocess (node/npx) │
│ stdin=PIPE, stdout=PIPE │
│ │
│ 2. Initialize (JSON-RPC 2.0): │
│ ──▶ {"method": "initialize", │
│ "params": { │
│ "protocolVersion": │
│ "2024-11-05"}} │
│ ◀── {"result": {serverInfo}} │
│ │
│ 3. Send initialized notification: │
│ ──▶ {"method": │
│ "notifications/initialized"} │
│ │
│ 4. Call tool: │
│ ──▶ {"method": "tools/call", │
│ "params": { │
│ "name": "web_search", │
│ "arguments": {...}}} │
│ ◀── {"result": {content: [...]}} │
│ │
│ 5. Terminate subprocess │
└────────┬────────────────────────────────┘
│
▼
┌─────────────────┐
│ Return Result │
│ to Frontend │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Add to Message │
│ tool_results[] │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Resubmit to LLM │
│ with results │
└─────────────────┘
4. SSE Event Broadcasting
┌─────────────────────────────────────────────────────────────────┐
│ BACKGROUND TASK │
│ _collect_and_broadcast_metrics() │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Every 5 seconds: │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │find_inference│ │ get_gpus() │ │scrape vLLM │ │
│ │ _process() │ │ │ │ /metrics │ │
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
│ │ │ │ │
│ └───────────────────┴───────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ Calculate rates │ │
│ │ Update lifetime │ │
│ │ Update peak │ │
│ └────────┬─────────┘ │
│ │ │
└────────────────────────────┼────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ EVENT MANAGER │
│ events.py │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Channels: │
│ "default" ──▶ status, gpu, metrics events │
│ "logs:{session_id}" ──▶ log line events │
│ │
│ ┌─────────────────┐ │
│ │ publish() │ │
│ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────┐ │
│ │ For each subscriber: │ │
│ │ │ │
│ │ ┌──────────┐ ┌──────────┐ ┌────────┐│ │
│ │ │ Queue 1 │ │ Queue 2 │ │Queue N ││ │
│ │ │(max 100) │ │(max 100) │ │ ││ │
│ │ └──────────┘ └──────────┘ └────────┘│ │
│ │ │ │
│ │ If queue full: drop oldest event │ │
│ └─────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ FRONTEND │
│ useSSE.ts │
├─────────────────────────────────────────────────────────────────┤
│ │
│ EventSource connection to /events │
│ │
│ onmessage: │
│ │ │
│ ├──▶ event.type = "status" │
│ │ └── Update running model, inference_ready │
│ │ │
│ ├──▶ event.type = "gpu" │
│ │ └── Update GPU cards display │
│ │ │
│ ├──▶ event.type = "metrics" │
│ │ └── Update throughput, cache, energy stats │
│ │ │
│ ├──▶ event.type = "launch_progress" │
│ │ └── Update progress bar (0-100%) │
│ │ │
│ └──▶ event.type = "log" │
│ └── Append to log viewer │
│ │
│ Reconnection: exponential backoff (1s, 2s, 4s, 8s, 16s) │
│ │
└─────────────────────────────────────────────────────────────────┘
Module Reference
Controller Core
| Module | Purpose | Key Functions |
|---|---|---|
app.py |
FastAPI app, lifespan, singletons | lifespan(), get_*_store() |
config.py |
Settings with env vars | Settings class |
models.py |
Pydantic data models | Recipe, MCPServer, ProcessInfo |
backends.py |
Build vLLM/SGLang commands | build_vllm_command(), build_sglang_command() |
process.py |
Process management | find_inference_process(), launch_model(), evict_model() |
store.py |
SQLite persistence | RecipeStore, ChatStore, MCPStore, PeakMetricsStore |
events.py |
SSE broadcasting | EventManager.publish(), subscribe() |
thinking_config.py |
Reasoning tokens | calculate_thinking_tokens() |
metrics.py |
Prometheus export | Counters, Gauges, Histograms |
gpu.py |
GPU info | get_gpu_list() |
Routes
| Route File | Endpoints | Purpose |
|---|---|---|
system.py |
/health, /status, /gpus, /config |
System info & health |
lifecycle.py |
/recipes/*, /launch/*, /evict |
Model lifecycle |
models.py |
/v1/models, /v1/studio/models |
OpenAI compatibility |
chats.py |
/chats/* |
Chat session CRUD |
logs.py |
/logs/*, /events |
Logs & SSE stream |
monitoring.py |
/metrics, /peak-metrics |
Prometheus & benchmarks |
usage.py |
/usage |
Analytics dashboard |
proxy.py |
Chat completions proxy | Auto-model-switching |
mcp.py |
/mcp/* |
MCP server & tool management |
Frontend
| Module | Purpose |
|---|---|
lib/api.ts |
APIClient with retry logic |
lib/types.ts |
TypeScript interfaces |
hooks/useSSE.ts |
SSE connection with reconnection |
hooks/useContextManager.ts |
Token tracking & compaction |
app/chat/page.tsx |
Main chat interface |
app/chat/utils/index.ts |
SSE parsing, helpers |
components/chat/* |
Chat UI components |
Configuration Reference
LiteLLM (config/litellm.yaml)
model_list:
- model_name: "model-alias"
litellm_params:
model: "openai/model-id"
api_base: "http://localhost:8000/v1"
litellm_settings:
callbacks:
- tool_call_handler # MCP integration
- prometheus_callback
cache: true
cache_params:
type: "redis"
ttl: 3600
Database Seeding
On first run, MCPStore._migrate() seeds:
- Exa Search server (if
exa-mcp-serveris installed globally)- Falls back to
npx -y exa-mcp-serverif not found - Requires
EXA_API_KEYenvironment variable
- Falls back to
Docker Deployment
Frontend Production Server
The frontend runs on port 3000 inside Docker. After making any frontend changes, you must rebuild the container:
# Rebuild and restart frontend container
docker compose up -d --build frontend
# Or rebuild all services
docker compose up -d --build
Important:
- The frontend is built at container creation time (not hot-reloaded)
- Changes to
frontend/src/**require a container rebuild - Running
npm run devlocally does NOT update the Docker container - Always verify changes are deployed by checking
docker compose ps
Service Ports
| Service | Container Port | Host Port |
|---|---|---|
| Frontend | 3000 | 3000 |
| Controller | 8080 | 8080 |
| LiteLLM | 4000 | 4100 |
| vLLM/SGLang | 8000 | 8000 |
| PostgreSQL | 5432 | 5432 |
| Redis | 6379 | 6379 |
| Prometheus | 9090 | 9090 |
Error Handling Patterns
Launch Failure Recovery
- Check process crashed → Read last 100 lines of log
- Timeout waiting for ready → Report timeout with log path
- Lock contention → Preempt existing launch after 2s
SSE Reconnection
- Connection lost → Exponential backoff retry
- Max reconnects reached → Show reconnect button
- Server restart → Auto-reconnect on next event
MCP Tool Errors
- Server not found → Return 404 with server ID
- Tool execution timeout → 30s timeout, return error
- JSON-RPC error → Extract message, return to model