vLLM Studio

Model lifecycle management for vLLM and SGLang inference servers.

What It Does

• Launch/evict models on vLLM or SGLang backends
• Save recipes - reusable model configurations with full parameter support
• Reasoning support - auto-detection for GLM (glm45), INTELLECT-3 (deepseek_r1), and MiniMax (minimax_m2_append_think) parsers
• Tool calling - native function calling with auto tool choice (auto-detected for GLM and INTELLECT-3 models)
• Web UI for chat, model management, and usage analytics
• LiteLLM integration for API gateway features (optional)

Architecture

┌──────────┐      ┌────────────┐      ┌─────────────┐
│  Client  │─────▶│ Controller │─────▶│ vLLM/SGLang │
│          │      │   :8080    │      │    :8000    │
└──────────┘      └────────────┘      └─────────────┘
                        │
                  ┌─────┴─────┐
                  │  Web UI   │
                  │   :3000   │
                  └───────────┘

Optional: Add LiteLLM as an API gateway for OpenAI/Anthropic format translation, cost tracking, and routing.

Quick Start

# Install controller deps
cd controller && bun install

# Run controller
cd controller && bun src/main.ts

# (Optional) Run frontend
cd frontend && npm install && npm run dev

# Or use the repo helper (starts optional Docker services too):
./start.sh --dev

API Reference

Health & Status

Recipes

Model Lifecycle

Chat Sessions

MCP (Model Context Protocol)

Configuration

Environment Variables

VLLM_STUDIO_PORT=8080           # Controller port
VLLM_STUDIO_INFERENCE_PORT=8000 # vLLM/SGLang port
VLLM_STUDIO_API_KEY=your-key    # Optional auth
VLLM_STUDIO_TEMPORAL_ADDRESS=localhost:7233 # Temporal server address

Recipe Example

{
  "id": "llama3-8b",
  "name": "Llama 3 8B",
  "model_path": "/models/Meta-Llama-3-8B-Instruct",
  "backend": "vllm",
  "tensor_parallel_size": 1,
  "max_model_len": 8192,
  "gpu_memory_utilization": 0.9,
  "trust_remote_code": true
}

All Recipe Fields

Directory Structure

vllm-studio/
├── controller/
│   └── src/main.ts    # Bun + Hono controller entrypoint
├── frontend/          # Next.js web UI
├── cli/               # CLI / TUI client
├── swift-client/      # SwiftUI client
├── shared/            # Shared TS types (wire shapes)
├── config/
│   └── litellm.yaml   # LiteLLM config (optional)
└── docker-compose.yml

With LiteLLM (Optional)

For OpenAI/Anthropic API compatibility:

docker compose up litellm

Then use http://localhost:4100 as your API endpoint with any OpenAI-compatible client.

Temporal (Local, OSS)

The Docker bundle includes a local Temporal dev server for workflows.

docker compose up temporal

Temporal listens on port 7233 (gRPC) and exposes the UI on port 8233.

License

Apache 2.0