Qwen DGX Spark: 20% more throughput with MTP1

Qwen3.6-35B-A3B-FP8 has been my local daily driver on the for a few weeks now NVIDIA DGX Spark. It takes on agentic tasks, delivers structured output and works stably with tools. I wanted to know whether there was still performance in serving without changing checkpoints or agent contracts.

In the lab I froze the checkpoint, hardware and measurement protocol. Then I changed the recipe step by step and tested throughput, tool calls, strict JSON and real text tasks. The fastest candidate managed almost 80 percent more tokens per second, but failed the tool evaluation. MTP1 was better in the end.

Under vLLM, TG128 throughput increased at the same time Benchy-Protocol from 53.8 to 64.6 tok/s: 20.0 percent more for decode-heavy agent requests. The time-to-first response increased from 112.7 to 148.2 ms. Tool-Eval Short remained at 100 out of 100, Hardmode at 91 out of 100; The production fixtures still showed limitations. The key question: Does the system remain reliable despite higher throughput?

The measurement contract

The protocol is in front of every number. Core ablation freezes checkpoint, container, hardware and benchmark configuration; only the number of MTP draft tokens changes. DFlash and NVFP4 later intentionally stand aside as complete alternative recipes, not as causal one-switch tests.

The recipe matrix on the DGX Spark

Five serving variants ran on the same machine. Baseline, MTP1 and MTP2 form the directly comparable nuclear ablation. DFlash adds a separate drafter and changes, among other things, KV data type, batch budget and concurrency; NVFP4/MTP3 also changes quantization format and container. Benchmark note: These two alternatives are only comparable to a limited extent. The performance values ​​come from AISyndicate’s own measurements from July 2026. The quality values ​​come from tool eval gates with native reasoning. They show the agent contract of the respective recipe, but do not form a strict quality ranking due to the additional stack differences in DFlash and NVFP4.

Decode throughput by output length. The solid lines represent direct seed stack ablation; DFlash and NVFP4/MTP3 are dashed as alternative recipes that are comparable to a limited extent. Error bars at TG128 show ± σ from three replicates.

The table shows the mean and standard deviation from three TG128 runs each. It is a recipe matrix on a machine, not a universal model ranking.

Baseline: Where the model stands without MTP

The FP8-Daily configuration serves Qwen3.6-35B-A3B-FP8 in native 262K context without speculative decoding. Over the four output lengths it remains almost flat at around 53.8 tok/s; the individual values ​​are in the matrix. The time-to-first response of 112.7 ms remains clean for a 35B checkpoint on 128 GB unified memory. This is the reference point against which every optimization is measured. Anything faster must maintain or improve this contract; anything that breaks it is not optimization.

MTP1: What triggers speculative decoding in the Qwen checkpoint

Multi-Token Prediction is already natively trained in the official Qwen3.6 checkpoint. No separate draft model, the draft head sits in the same checkpoint. In vLLM, the speculative config is activated with the MTP method and a draft token; The full command is in the recipe below. The model suggests one draft token per step, the target model verifies it in the same forward pass. If the acceptance rate is high, this saves a complete decode step.

On the TG128 profile with auto backend, the throughput jumps from 53.8 to 63.1 tok/s. The effect is noticeable across all output lengths, but loses relative strength as the sequence length increases: TG32 increases the most, TG512 the least. That is to be expected. Speculative decoding is particularly worthwhile when the overhead per token remains low in relation to the number of passes saved. All metrics in the following output matrix: AISyndicate proprietary benchmarks, ASUS GX10 / DGX Spark, July 2026.

For agents with typical response lengths between 100 and 300 tokens, MTP1 ablation is the sweet spot. Those who prefer to generate long reports see smaller profits, but still noticeable double-digit percentages.

A second draft token does not improve this curve. MTP2 achieves 67.6 tok/s with TG32 and is therefore 2.6 percent ahead of MTP1. However, for TG128, throughput drops to 60.1 tok/s, 7.0 percent below MTP1; at TG256 it remains practically the same and at TG512 it drops by 4.9 percent. At the same time, the TTFR increases by 20.4 to 28.7 percent compared to MTP1 across all four output lengths. Tool-Eval Short remains completely green at 100 out of 100. The agent contract holds; The serving still gets worse. This fits with the vLLM boot warning that repeated use of the same MTP layer can lower the adoption rate. We did not measure them separately. On this stack, the sweet spot is a draft token.

FlashInfer pinning is reproducibility control, not miracle gain

This is where the most common misinterpretation lurks. vLLM had already selected FlashInfer in auto mode. The pinned run on TG128 is 2.5 percent higher than MTP1-Auto (64.6 instead of 63.1 tok/s), but loses slightly on TG256 and TG512. This is not a resilient second speedup. The actual jump comes from MTP1; Pinning primarily fixes the backend identity against shifted defaults between container builds.

FlashInfer provides paged attention kernels like BatchDecodeWithPagedKVCacheWrapper ready. On the DGX Spark it was visible in the bootlog as an active attention backend. The 20 percent from the target thesis cannot be attributed to FlashInfer in isolation. You measure the delta between baseline and the final MTP1 recipe. The pin makes this runtime path reproducible.

TTFR and cold start: The conscious trade-off

Speculative decoding has costs. The time-to-first response increases from 112.7 ms to 148.2 ms. This is noticeable for streaming chats. In our agentic profile, the price was acceptable because the higher decode throughput reduces the overall latency of longer responses; However, the additional TTFR can remain relevant for short tool calls.

During the cold start there were two observations that are part of the protocol. Two MTP cold starts each showed an incorrect Strict JSON response on the first attempt; an immediate identical retry provided valid structure. The production cold start with the promoted recipe remained valid on the first try. Short and hardmode suites did not reproduce the error. The sample is small (n=2+1). From these three observations, we treat this as a variance signal in the very first request for model load without a final cause analysis and document it transparently instead of smoothing it.

Quality Gates: Why DFlash was kicked out

DFlash and the associated FlashQLA Attention variant delivered spectacular raw numbers: 102.7, 96.7, 86.3 and 69.8 tok/s across the four output lengths. That’s 79.6 percent more throughput on TG128 than the baseline. Anyone who only measures decode speed would immediately promote the candidate.

We measure agentically: choose tools, structure arguments, continue calls, handle errors and refuse actions. The Tool Eval Bench includes 15 scenarios and 30 points. The winner scores 100 out of 100; DFlash three times 53 out of 100, with structural breaks and garbled tool arguments.

The mechanical length contract was repeated as a timely A/B with three runs per FP8 lane. The pre-MTP baseline achieved 22/24, MTP1 20/24 usable short texts; the teasers passed 24/24 on both sides. All errors were purely length overruns. The small baseline lead is a warning signal, but is not evidence of a causal or significant MTP regression in this sample.

The second level is editorial: in both FP8 lanes we found unsubstantiated technical or causal details. When faced with thin input, MTP1 was more likely to respond confidently, while the baseline was more likely to refrain from responding. This is a directional signal, not a statistically proven loss of quality. Tool-Eval 100/100 remains strong, but does not replace evidence filters or human review.

NVFP4 achieved 5/8 on the same eight-case set in an older run. Because quantization, containers and serving switches changed together, this value does not belong in the direct FP8-A/B. Structured JSON scoring and source-linked research note extraction also ran on several recipes; different source code lengths prevent pure latency A/B.

NVFP4/MTP3 is an alternative recipe, not A/B

The NVIDIA recipe works NVFP4 quantization with three Speculative tokens in another container. TG128 is at 114.0 tok/s, but with a significantly higher dispersion than MTP1. It’s not an isolated A/B: quantization format, containers and other serving switches change together. Tool-Eval Hardmode remained strong at 89 out of 100; However, in separate production-related tests, short factual texts were less reliable. That’s why the recipe remained an alternative lane instead of our daily profile.

Hardware compatibility

The measured values ​​only apply to the tested DGX Spark. For other hardware, the following matrix separates reliable statements from mere transferability.

The following YAML maps the reproducible serving contract behind all the numbers in the text. Network address and port are intentionally generic. Anyone who runs the configuration on a DGX Spark with a comparable system load should end up within the range of our repetitions.

recipe_version: "1"
name: qwen3.6-35b-a3b-fp8-256k-vllm-freshwheels-reasoning-c4-mtp1-flashinfer-pinned
description: Qwen3.6 FP8 256K MTP1 with explicitly pinned FlashInfer attention
model: Qwen/Qwen3.6-35B-A3B-FP8
solo_only: true
container: vllm-node-freshwheels-20260713

defaults:
  port: 8000
  host: 0.0.0.0
  served_model_name: Qwen/Qwen3.6-35B-A3B-FP8
  max_model_len: 262144
  gpu_memory_utilization: 0.70
  max_num_batched_tokens: 16384
  max_num_seqs: 4

env:
  VLLM_ENABLE_CUDAGRAPH_GC: 1
  VLLM_USE_FLASHINFER_SAMPLER: 1
  VLLM_USE_DEEP_GEMM: 0

command: |
  vllm serve Qwen/Qwen3.6-35B-A3B-FP8 
    --served-model-name {served_model_name} 
    --quantization fp8 
    --language-model-only 
    --max-model-len {max_model_len} 
    --gpu-memory-utilization {gpu_memory_utilization} 
    --max-num-batched-tokens {max_num_batched_tokens} 
    --max-num-seqs {max_num_seqs} 
    --kv-cache-dtype fp8 
    --enable-prefix-caching 
    --enable-auto-tool-choice 
    --tool-call-parser qwen3_coder 
    --reasoning-parser qwen3 
    --speculative-config '{"method":"mtp","num_speculative_tokens":1}' 
    --attention-backend flashinfer 
    --port {port} 
    --host {host}

What remains

Serving recipes are not fine-tuning cosmetics; they decide whether LLM inference on local hardware is actually worthwhile for agents. MTP1 delivers 20 percent more TG128 throughput and keeps the generic tool gates stable. The production fixtures also show that length and evidence filters as well as human reviews remain necessary. DFlash is the drastic warning, MTP1 is the useful optimization with limits.

The recipe is above, have fun measuring and experimenting!