vLLM Studio Controller

TypeScript backend for vLLM Studio, built on Bun + Hono + SQLite. Manages model lifecycle (launch/evict/preempt) across vLLM, SGLang, llama.cpp, and TabbyAPI backends, with real-time monitoring, agent runtime, and OpenAI-compatible API proxying.

Quick Start

# Install dependencies
bun install

# Run (development with auto-reload)
bun --watch src/main.ts

# Run (production)
bun src/main.ts

# Type check
npx tsc --noEmit

# Lint + format
bun run lint:fix && bun run format

# Tests
bun test

Important: Use native bun (~/.bun/bin/bun), not snap bun. Snap bun has sandbox restrictions that block nvidia-smi, breaking GPU monitoring. Use ./start.sh from the project root which handles this automatically.

Architecture

main.ts                    Entry point, Bun.serve(), shutdown handlers
  └── app-context.ts       Dependency injection container (AppContext)
        ├── config/env.ts   Environment config (.env cascade)
        ├── core/           Logger, errors, AsyncLock, log files
        ├── http/app.ts     Hono app, CORS, route registration
        ├── routes/         HTTP endpoints (16 route modules)
        ├── services/       Business logic (process mgmt, backends, GPU, metrics, agent runtime)
        └── stores/         SQLite persistence (recipes, chats, MCP, metrics, downloads)

Request Flow

HTTP Request → Hono middleware (CORS, auth, error handling)
  → Route handler (routes/*.ts)
    → Service layer (services/*.ts)
      → Store layer (stores/*.ts) → SQLite
      → Process manager → spawn/kill inference processes
      → Event manager → SSE broadcast to connected clients

Dependency Injection

All components receive AppContext, created once in app-context.ts:

interface AppContext {
  config: RuntimeConfig;
  logger: Logger;
  eventManager: EventManager;
  launchState: LaunchState;
  metrics: ControllerMetrics;
  metricsRegistry: MetricsRegistry;
  processManager: ProcessManager;
  downloadManager: DownloadManager;
  runManager: ChatRunManager;
  stores: {
    recipeStore: RecipeStore;
    chatStore: ChatStore;
    downloadStore: DownloadStore;
    peakMetricsStore: PeakMetricsStore;
    lifetimeMetricsStore: LifetimeMetricsStore;
    mcpStore: McpStore;
  };
}

Route Registration

All routes follow the same pattern and are wired in http/app.ts:

registerSystemRoutes(app, context);
registerLifecycleRoutes(app, context);
registerMonitoringRoutes(app, context);
// ...etc

Source Structure

src/
├── main.ts                  Server startup, nvidia-smi check, shutdown
├── app-context.ts           DI container creation
├── metrics-collector.ts     Background metrics loop (5s interval)
│
├── config/
│   ├── env.ts               RuntimeConfig, .env loading with cascade
│   └── persisted-config.ts  Persistent settings (SQLite-backed)
│
├── core/
│   ├── logger.ts            Log levels, disk + SSE output
│   ├── errors.ts            HttpStatus class (notFound, badRequest, etc.)
│   ├── async.ts             AsyncLock (mutex with timeout/cancellation), AsyncQueue, delay
│   └── log-files.ts         Log rotation, cleanup, path helpers
│
├── types/
│   ├── models.ts            Backend, Recipe, ProcessInfo, LaunchResult, GpuInfo
│   ├── context.ts           AppContext interface
│   ├── schemas.ts           Zod validation schemas
│   └── brand.ts             Branded types
│
├── http/
│   ├── app.ts               Hono app setup, middleware, route registration
│   └── sse.ts               SSE streaming helpers
│
├── routes/
│   ├── system.ts            /health, /status, /gpus, /config
│   ├── lifecycle.ts         /recipes CRUD, /launch/{id}, /evict, /wait-ready
│   ├── models.ts            /v1/models (OpenAI format), model discovery
│   ├── chats.ts             /chats CRUD, messages, forking
│   ├── openai.ts            OpenAI-compatible chat completions proxy
│   ├── logs.ts              /logs list/stream, /events SSE
│   ├── monitoring.ts        /metrics (Prometheus), /peak-metrics, /benchmark
│   ├── usage.ts             /usage analytics (Postgres-backed)
│   ├── mcp.ts               /mcp/servers, /mcp/tools
│   ├── downloads.ts         Model download endpoints
│   ├── runtime.ts           Runtime info (versions, paths)
│   ├── studio.ts            Studio-specific endpoints
│   ├── tokenization.ts      /tokenize endpoint
│   └── agent-files.ts       Agent filesystem operations
│
├── services/
│   ├── process-manager.ts   Process spawn, findInferenceProcess, eviction
│   ├── backends.ts          Command builders for vLLM, SGLang, llama.cpp, TabbyAPI
│   ├── event-manager.ts     SSE publish/subscribe with channels
│   ├── gpu.ts               nvidia-smi wrapper, memory estimation
│   ├── metrics.ts           Prometheus registry (prom-client)
│   ├── model-browser.ts     Local model directory scanning
│   ├── download-manager.ts  HuggingFace model downloads
│   ├── launch-state.ts      Launch state tracking (mutex, abort)
│   ├── llamacpp-runtime.ts  llama.cpp-specific runtime handling
│   ├── vllm-runtime.ts      vLLM-specific runtime handling
│   ├── runtime-info.ts      System/runtime info gathering
│   ├── chat-compaction.ts   Chat message compaction
│   ├── tool-call-core.ts    Tool call parsing/execution
│   ├── proxy-parsers.ts     Response stream parsing
│   ├── process-utilities.ts Process helper functions
│   ├── command/             Command building utilities
│   ├── downloads/           Download helpers (globs, math, paths, HF API)
│   └── agent-runtime/       Agent orchestration system
│       ├── run-manager.ts          Chat run orchestration
│       ├── agent-event-handler.ts  Event handling for agent runs
│       ├── message-mapper.ts       OpenAI <-> Agent message mapping
│       ├── run-manager-persistence.ts  Run persistence
│       ├── run-manager-sse.ts      Run SSE streaming
│       ├── system-prompt-builder.ts  System prompt construction
│       ├── tool-registry.ts        Tool registry interface
│       ├── tool-registry-common.ts Common tools (web search, etc.)
│       ├── tool-registry-agentfs.ts AgentFS integration
│       ├── tool-registry-mcp.ts    MCP tool integration
│       ├── tool-registry-plan.ts   Planning tool
│       ├── stream-openai-completions-safe.ts  Safe streaming
│       └── model-factory.ts        Model instance creation
│
└── stores/
    ├── recipe-store.ts      Recipe CRUD + migrations
    ├── chat-store.ts        Chat sessions/messages
    ├── chat-store-schema.ts Schema definitions
    ├── chat-store-hydration.ts Data hydration
    ├── chat-store-runs.ts   Run persistence
    ├── mcp-store.ts         MCP server configuration
    ├── download-store.ts    Download tracking
    ├── metrics-store.ts     PeakMetricsStore + LifetimeMetricsStore
    └── recipe-serializer.ts Recipe JSON serialization

Key Services

Process Manager (services/process-manager.ts)

Spawns inference backends as detached child processes. Monitors process health via PID liveness checks. Handles eviction with SIGTERM -> SIGKILL escalation.

Backends (services/backends.ts)

Builds CLI commands for each backend. Handles:

• vLLM: tensor parallelism, quantization, tool/reasoning parser auto-detection
• SGLang: similar to vLLM with SGLang-specific flags
• llama.cpp: GGUF model loading, split modes (row/layer/graph), flash attention, KV cache quantization
• TabbyAPI: ExLlamaV2-based serving

Launch Lifecycle (routes/lifecycle.ts)

State machine for model launches with:

• Preemption: launching a new model auto-evicts the current one
• Cancellation: abort via /evict during launch
• PID liveness: detects dead processes within one poll cycle (~2s)
• Fatal pattern detection: catches argparse errors, import failures, tracebacks
• Progress events: SSE updates during model loading
• AsyncLock: serialized access prevents concurrent launches

Metrics Collector (metrics-collector.ts)

Background loop (5s interval) that:

• Scrapes inference backend /metrics endpoint (vLLM Prometheus metrics)
• Calculates token throughput from deltas
• Tracks GPU power consumption and energy (watt-hours)
• Updates peak metrics (best observed TPS per model)
• Publishes real-time SSE events (status, gpu, metrics)

Agent Runtime (services/agent-runtime/)

Multi-turn agent system with:

• Tool calling (MCP servers, AgentFS, web search, planning)
• Streaming OpenAI completions with safe parsing
• System prompt construction with context
• Run persistence and resumption

Databases

Two SQLite databases (via Bun's built-in driver):

Monitoring

Prometheus Metrics (prefix: vllm_studio_)

SSE Events (GET /events)

Configuration

Environment variables (all prefixed with VLLM_STUDIO_):

Development

# Watch mode (auto-reload on changes)
bun --watch src/main.ts

# Type check
npx tsc --noEmit

# Lint
bun run lint

# Format
bun run format

# Run tests
bun test

# Check for unused exports and duplicate code
bun run check

Adding a New Route

1. Create src/routes/my-feature.ts
2. Export registerMyFeatureRoutes(app: Hono, context: AppContext)
3. Register in src/http/app.ts

Adding a New Store

1. Create src/stores/my-store.ts extending the SQLite pattern
2. Initialize in app-context.ts
3. Add to AppContext.stores in types/context.ts