Quality — methodology, ablation, and delta recovery

The headline tables are in the README. This is the detailed companion.

What we read and compare to

Baseline = the untouched official DeepSeek‑V4‑Flash checkpoint (~149 GiB), verified from its safetensors headers: MoE experts are FP4 (I8, e2m1 packed 2/byte) with F8_E8M0 block‑32 scales (layers.0..42.ffn.experts.E.{w1,w2,w3}.weight, 33,024 tensors); dense and attention are FP8 (F8_E4M3, block‑128, per config.json). This project changes only the expert codes (FP4 → 2‑bit symmetric) — scales, dense FP8, and headers stay byte‑identical — and runs on the same serving stack (only MODEL_DIR differs).

Probe protocol (tools/probe_quant_quality.py)

1. MTP acceptance — mean accepted draft length + avg draft‑acceptance rate over 3×512‑token greedy generations (ignore_eos). Primary fidelity proxy (drafter+target share the model, so a degraded target lowers acceptance). Noise band ≈ ±0.05.
2. Coherence — 12 fixed prompts, exact‑match vs the official‑checkpoint outputs + manual verdict.
3. Arithmetic — 5 multi‑step problems (the official model itself tops out at 3/5).
4. Long context — passphrase needle at 100K/250K/440K depths (tools/needle_probe.py).

Bits‑vs‑quality ablation (same stack, only the expert codes change)

The finding: it was sign bias, not magnitude

Naive 2‑bit collapses to a degenerate loop (}<?}<?…). The optimal‑L2 2‑bit codebook goes sign‑asymmetric (drops one sign's tail, e.g. {−3,−1,0.5,4}); accumulated per‑expert sign bias over 43 layers destroys the model. Forcing a sign‑symmetric codebook at the same L2 error restores MTP acceptance to at/above the official FP4 experts (2.73/86.3 vs 2.68/84.1), 12/12 coherent. Symmetry — not granularity — is the lever (33,023/33,024 tensors pick the same {−4,−1,1,4}).

Delta recovery (FP4 hot‑expert tier)

Symmetric 2‑bit already matches the official experts on acceptance + coherence. The residual is numerical precision (multi‑step arithmetic) — the 2‑bit base is "coherent but numerically sloppy." Caching the hot experts at FP4 closes it:

Production 1‑GPU end‑to‑end A/B (warmed): MTP acceptance with the delta tier 2.725 / 2.744 (mean/median, σ 0.136) — at/above the same‑run two‑card FP4 control 2.546 / 2.496 (σ 0.172).

Studying the precision tiering

The delta tier is a workload‑adaptive mixed‑precision scheme — the set of FP4 experts is chosen at runtime from routing. The manager is instrumented so this can be measured directly (default off; logging only, no effect on the serving path):

VLLM_MOE_W2_DELTA_TRACE=1                 # coverage + churn summary + per-layer FP4 histogram
VLLM_MOE_W2_DELTA_TRACE=2                 # + one line per promote/evict event
VLLM_MOE_W2_DELTA_TRACE_EVERY=64          # ticks between summaries
VLLM_MOE_W2_DELTA_DUMP=/tmp/delta.json    # atomic precision-map snapshots (tail-able)

Each summary reports filled slots, coverage (cached / layers×experts), per‑window and cumulative promote/evict counts, and FP4 experts per layer; the JSON dump is the full {layer: [fp4 expert ids]} map plus counters. DeltaTier.precision_of(layer, expert) and precision_map() expose the same state programmatically. tools/delta_trace_demo.py drives the real manager with synthetic Zipfian routing so you can see this output (and validate the instrumentation) in ~1 s without loading the model. Questions this makes answerable:

• Coverage vs. pool size — sweep VLLM_MOE_W2_DELTA_GB and watch how coverage % and the arithmetic/recall probes move; find the knee where extra VRAM stops buying quality.
• Working‑set stability — cumulative evictions / promotions over a long run is a churn rate; low churn means routing is concentrated and a small pool suffices.
• Per‑layer concentration — the histogram shows whether some layers route to far more distinct experts than others (where precision matters most).

These are open questions the design invites; the README tables are the headline, this is the instrumentation to push further.

Reproduce

python3 tools/probe_quant_quality.py <port>           # acceptance + coherence + arithmetic
python3 tools/needle_probe.py <port> 48000 0.1        # long-context retrieval
# ablation variants are produced offline by tools/repack_expert_bits.py (quality probe only)

These are engineering‑grade fidelity checks, not a standardized benchmark sweep (MMLU/GSM8K‑style evals are a natural next step). Discipline: every surprising delta is re‑measured against a same‑GPU placebo control before it's believed.