Worker vs. Verifier Latency Benchmark Report

1. Executive Summary

Core conclusion: the verifier's cost is far below the worker's generation cost — in typical multi-token output scenarios, verifying a full generation takes only about 1% of the cost to produce it (worker/verifier ≈ 80×–117×). This holds for both Qwen3-8B (RTX 4090) and Qwen3-32B (H100 NVL), i.e. it is robust to model size and hardware. This is direct evidence that the worker/verifier protocol is viable in terms of time cost.

The gap stems from the fundamentally different nature of the two paths. The worker spends nearly all its time on serial decode, emitting tokens one at a time — ~17.5 ms/token for the 8B on a 4090, and ~22.7 ms/token for the 32B on an H100 NVL — essentially independent of input length. The verifier performs a single parallel prefill, computing all positions of input+output at once at several thousand tok/s. The ~two-orders-of-magnitude efficiency gap between serial decode and parallel prefill is the source of the cost difference. Consequently, the longer the output, the worse the worker fares and the more favorable verification becomes: the ratio peaks around 100×–117× when output is large.

Notably, the 32B model is only ~30% slower per decode token than the 8B despite being ~4× larger, because the H100 NVL's much higher memory bandwidth largely offsets the extra weight traffic. The verifier's parallel prefill throughput is comparable across the two setups, so the overall worker/verifier ratio lands in the same band.

The only exception is when the output is extremely short (=1), where the verifier is actually slightly slower (ratio 0.62–0.90) — it must compute top-k logprobs at every position, and that fixed overhead slightly outweighs the tiny decode savings. This is a corner case and does not affect the main conclusion.

2. Worker Cost Model: Output-Dominated, Serial Decode

With input fixed at 128, worker time scales almost perfectly linearly with output, and the per-token cost holds steady — ~17.5 ms for the 8B, ~22.7 ms for the 32B:

Input has little effect on the worker — with output fixed at 1024, growing input from 128 to 8192 raises worker time by only ~10–12%:

The pure prefill cost can be read approximately from the output=1 rows (8B: input=8192 / out=1 → 0.886s; 32B: input=8192 / out=1 → 1.428s).

Conclusion: worker cost ≈ output_len × (per-token decode cost), consistent with decode being serial and memory-bandwidth-bound. The larger model pays more per token, but only modestly, thanks to the faster GPU's bandwidth.

3. Verifier Cost Model: Total-Dominated, Parallel Prefill

The verifier is a single prefill; its time scales linearly with total (input+output). Effective throughput is a few thousand tok/s for both setups:

Conclusion: verifier cost ≈ total_tokens / (a few thousand tok/s) + fixed overhead. Prefill computes all positions in one parallel pass, making it two orders of magnitude faster than serial decode. The 32B verifier is only moderately slower than the 8B's despite the larger weights, again because the H100 NVL absorbs the extra bandwidth demand.

4. Key Metric: Worker/Verifier Cost Ratio

This is the most valuable output of the experiment. The ratio grows sharply with output before flattening. Representative scenarios for both models:

¹ The 32B 128/2048 cell is a noisy outlier: its verifier median (0.630s) sits above the mean (0.559s) and even the p10 (0.424s), so the ratio there is depressed by verifier-side jitter rather than a real effect.

4.1 Qwen3-8B (RTX 4090) — full ratio matrix

4.2 Qwen3-32B (H100 NVL) — full ratio matrix

How to read the tables (both matrices show the same structure):

• The output=1 column is entirely < 1: the verifier is slightly slower than the worker due to the per-position logprobs overhead.
• Each row rises monotonically with output: the longer the output, the more cost-effective verification becomes.
• Each column decreases as input grows: with longer input, the verifier must pay for the extra input-segment prefill, narrowing the ratio.
• For a fixed input the ratio rises then flattens/dips slightly: when output ≫ input, the verifier's total ≈ output also scales ∝ output, so the ratio asymptotes to (per-token decode) / (per-token prefill) ≈ 90–110×.
• The two models track each other closely: same shape, same magnitude, peaks in the 100×–117× range. The worker/verifier advantage is not an artifact of one particular model or GPU.

5. Notes and Limitations

• The TTFT / decode split columns are empty (-). Both runs used the vLLM V1 engine (the EngineCore in the logs), which does not expose the legacy RequestMetrics.first_token_time, so the script cannot separate TTFT from decode. This does not affect the total timings or any conclusion above; for a prefill approximation, use the output=1 rows.
• This is single-request latency at batch=1. In real online serving, the worker amortizes per-token cost via continuous batching, and the verifier can verify in batches, so absolute numbers will change. However, the fundamental difference between prefill (parallel) and decode (serial) remains, so the order-of-magnitude relationship — "verification is far cheaper than generation" — still holds.
• Cross-run comparison caveat. The 8B and 32B runs use different GPUs (RTX 4090 vs H100 NVL), so per-model absolute latencies are not directly comparable; what *is* comparable, and what this report emphasizes, is the worker/verifier *ratio* within each run.
• A few individual cells are noisy (notably 32B 128/2048, see §4). Verifier times for small totals are dominated by fixed overhead and are more susceptible to jitter; treat single-cell anomalies with caution and rely on the overall trend.

6. Reproduction

Qwen3-8B (RTX 4090):

cd tools/vllm_worker_verify_pipeline
./benchmark_worker_verifier_latency.py \
  --model Qwen/Qwen3-8B \
  --dtype bfloat16 --trust-remote-code \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --input-lengths "128 512 1024 2048 4096 8192" \
  --output-lengths "1 16 64 256 1024 2048 4096 8192" \
  --top-k 20 --warmup 2 --repeats 5 \
  --gpu "4090" \
  --output-dir qwen3_8b

Qwen3-32B (H100 NVL):

cd tools/vllm_worker_verify_pipeline
./benchmark_worker_verifier_latency.py \
  --model Qwen/Qwen3-32B \
  --dtype bfloat16 --trust-remote-code \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --input-lengths "128 512 1024 2048 4096 8192" \
  --output-lengths "1 16 64 256 1024 2048 4096 8192" \
  --top-k 20 --warmup 2 --repeats 5 \
  --gpu "h100nvl" \
  --output-dir qwen3_32b

(These are the actual arguments that produced the committed data; the results are stored under experiments/worker-verifier-latency/data/qwen3_8b/ and experiments/worker-verifier-latency/data/qwen3_32b/. Note --max-model-len 32768 is required — the largest cells reach total = 16384 tokens, which an 8192 context cannot hold. The --gpu value is a free-form label recorded verbatim into metadata.json, not a detected device.)

Artifacts (per model dir): latency_summary.csv (aggregated metrics), latency_raw.jsonl (per-iteration raw timings), summary.md (Markdown table), metadata.json (args and environment).

Appendix A: Full Data Table — Qwen3-8B (RTX 4090)

Appendix B: Full Data Table — Qwen3-32B (H100 NVL)