Qwen3.6 27B & 35B on vLLM: Single Radeon R9700 Tuning Results

A r/LocalLLaMA post details how to run Qwen3.6 27B (dense) and 35B (MoE) on a single Radeon AI Pro R9700 using vLLM Radiance via Podman. The author shares their exact configuration and benchmark results, which are particularly useful for AMD GPU users.

Setup and Key Differences

They use the stilldeadcode/vllm-radiance:0.5.8 container, which ships with a reference config tuned for FP8 weights on dual R9700s with tensor parallelism (TP=2). For a single card with INT4 weights, the following changes are needed:

• --tensor-parallel-size 1 (no second card)
• --gpu-memory-utilization 0.98 (reference was 0.90–0.97)
• num_speculative_tokens=4 on the 27B — ladder-tested 2/3/4/8, with 4 winning by 17–48% over 8 at all depths

The weights are Avesed/Qwen3.6-{27B,35B}-INT4-W4A16 (compressed-tensors, group_size 32). The 35B at FP8 simply won't fit on a 32GB card at useful context lengths.

Critical Fix: tokenizer.json

The Avesed INT4 repo ships a tokenizer.json with truncation.max_length set to 512 and padding as Fixed(512) — likely from calibration. This breaks vision above ~672px. The fix: set both to null.

Benchmark Results

35B-A3B MoE (KV pool tokens = 440,241)

Depth	Prefill tok/s	Decode tok/s
4k	~7,800	61.4
16k	~7,700	60.1
50k	~6,040	57.0
78k	~5,120	54.7
100k	~4,580	52.9
150k	~3,690	49.5

27B dense, MTP spec=4 (KV pool tokens = 212,147)

Depth	Prefill tok/s	Decode tok/s	Mean accepted len
4k	~1,288	59.6	4.4
16k	~1,345	62.3	4.6
50k	~1,207	59.6	4.5
100k	~1,027	53.7	4.5

Notable: the 35B MoE achieves much higher prefill throughput (up to 7.8k tok/s vs 1.3k) but similar decode speeds. The 27B's MTP speculation delivers an accepted length of ~4.5 tokens.

This is a practical configuration for AMD users who don't have dual GPUs and want to run these models locally. The author is willing to provide startup scripts on request.