Standard autoregressive generation forces serving systems to commit tokens sequentially, turning memory bandwidth into your primary latency bottleneck. Speculative decoding bypasses this bottleneck by drafting candidate tokens with a lightweight component and verifying them simultaneously within a single forward pass of the target model.
Recent benchmarks from the vLLM team on AMD Instinct MI300X and MI355X GPUs demonstrate how radically throughput depends on the drafting architecture. Testing across native Multi-Token Prediction, Gemma MTP, EAGLE-3, DFlash, and DSpark reveals that longer draft proposals do not automatically produce faster serving.
If the acceptance rate drops even slightly, verification overhead wipes out the gains. Real speedups require balancing draft checkpoint size against the compute capacity of ROCm kernels. Parallel drafting methods like DFlash excel on code and structured text, whereas sequential draft models handle conversational tasks with fewer rollbacks.
Serving performance is no longer just about raw FLOPS; it is about maximizing verified tokens per memory access.

