vLLM Studio

Model lifecycle management for vLLM and SGLang inference servers.

What It Does

• Launch/evict models on vLLM or SGLang backends
• Save recipes - reusable model configurations with full parameter support
• Reasoning support - auto-detection for GLM (glm45), INTELLECT-3 (deepseek_r1), and MiniMax (minimax_m2_append_think) parsers
• Tool calling - native function calling with auto tool choice (auto-detected for GLM and INTELLECT-3 models)
• Web UI for chat, model management, and usage analytics
• LiteLLM integration for API gateway features (optional)

Architecture

┌──────────┐      ┌────────────┐      ┌─────────────┐
│  Client  │─────▶│ Controller │─────▶│ vLLM/SGLang │
│          │      │   :8080    │      │    :8000    │
└──────────┘      └────────────┘      └─────────────┘
                        │
                  ┌─────┴─────┐
                  │  Web UI   │
                  │   :3000   │
                  └───────────┘

Optional: Add LiteLLM as an API gateway for OpenAI/Anthropic format translation, cost tracking, and routing.

Quick Start

# Install controller
pip install -e .

# Run controller
vllm-studio

# (Optional) Run frontend
cd frontend && npm install && npm run dev

API Reference

Health & Status

Recipes

Model Lifecycle

Chat Sessions

MCP (Model Context Protocol)

Configuration

Environment Variables

VLLM_STUDIO_PORT=8080           # Controller port
VLLM_STUDIO_INFERENCE_PORT=8000 # vLLM/SGLang port
VLLM_STUDIO_API_KEY=your-key    # Optional auth

Recipe Example

{
  "id": "llama3-8b",
  "name": "Llama 3 8B",
  "model_path": "/models/Meta-Llama-3-8B-Instruct",
  "backend": "vllm",
  "tensor_parallel_size": 1,
  "max_model_len": 8192,
  "gpu_memory_utilization": 0.9,
  "trust_remote_code": true
}

All Recipe Fields

Directory Structure

vllm-studio/
├── controller/
│   ├── app.py         # FastAPI endpoints
│   ├── process.py     # Process management
│   ├── backends.py    # vLLM/SGLang command builders
│   ├── models.py      # Pydantic models
│   ├── store.py       # SQLite storage
│   ├── config.py      # Settings
│   └── cli.py         # Entry point
├── frontend/          # Next.js web UI
├── config/
│   └── litellm.yaml   # LiteLLM config (optional)
└── docker-compose.yml

controller-new (Bun canary, optional)

controller-new/ is an experimental Bun-based canary that keeps the controller as a full OpenAI /v1/* endpoint while allowing per-endpoint upstream routing.

Use it when you want to route different OpenAI surfaces to different backends, e.g.:

• route /v1/chat/* to a remote vLLM server
• route /v1/audio/* to a transcription service
• route /v1/images/* to an image generator

See controller-new/README.md.

With LiteLLM (Optional)

For OpenAI/Anthropic API compatibility:

docker compose up litellm

Then use http://localhost:4100 as your API endpoint with any OpenAI-compatible client.

License

Apache 2.0