P-EAGLE: Quicker LLM inference with Parallel Speculative Decoding in vLLM

EAGLE is the state-of-the-art technique for speculative decoding in giant language mannequin (LLM) inference, however its autoregressive drafting creates a hidden bottleneck: the extra tokens that you simply speculate, the extra sequential ahead passes the drafter wants. Finally these overhead eats into your positive aspects. P-EAGLE removes this ceiling by producing all Ok draft tokens in a single ahead go, delivering as much as 1.69x speedup over vanilla EAGLE-3 on actual workloads on NVIDIA B200.

You’ll be able to unlock this efficiency acquire by downloading (or coaching) a parallel-capable drafter head, including “parallel_drafting”: true on you vLLM serving pipeline. Pre-trained P-EAGLE heads are already obtainable on HuggingFace for GPT-OSS 120B, GPT-OSS 20B, and Qwen3-Coder 30B, so you can begin right this moment.

On this publish, we clarify how P-EAGLE works, how we built-in it into vLLM ranging from v0.16.0 (PR#32887), and the way to serve it with our pre-trained checkpoints. Right here is the record of artifacts used:

Fast begin P-EAGLE:

You’ll be able to allow parallel drafting with a single configuration change within the SpeculativeConfig class:

# vllm/config/speculative.py
   parallel_drafting: bool = True

Right here’s an instance command in vLLM to allow parallel drafting with P-EAGLE as drafter:

  vllm serve openai/gpt-oss-20b 
   --speculative-config '{"technique": "eagle3", "mannequin": "amazon/gpt-oss-20b-p-eagle", "num_speculative_tokens": 5, "parallel_drafting": true}'

EAGLE’s Drafting Bottleneck

EAGLE achieves 2–3× speedups over customary autoregressive decoding and is broadly deployed in manufacturing inference frameworks together with vLLM, SGLang, and TensorRT-LLM.EAGLE drafts tokens autoregressively. To supply Ok draft tokens, it requires Ok ahead passes by way of the draft mannequin. As drafter fashions get higher at drafting lengthy outputs, this drafting overhead turns into vital—the drafter’s latency scales linearly with hypothesis depth, constraining how aggressively we are able to speculate.

Our Strategy: Parallel-EAGLE (P-EAGLE)

We current P-EAGLE, which transforms EAGLE from autoregressive to parallel draft era. On B200 GPUs, P-EAGLE achieves 1.05×–1.69× speedup over vanilla EAGLE-3 on GPT-OSS 20B over MT-Bench, HumanEval, and SpeedBench. It’s now built-in into vLLM to unlock parallel speculative decoding, and able to speed up real-world deployments.

P-EAGLE generates the Ok draft tokens in a single ahead go. Determine 2 reveals the structure, which consists of two steps.

Step 1: Prefilling. The goal mannequin processes the immediate and generates a brand new token, as it might throughout regular inference. Alongside the way in which, P-EAGLE captures the mannequin’s inside hidden states: h_prompt for every immediate place, and h_context for the newly generated token. These hidden states encode what the goal mannequin “is aware of” at every place and can information the drafter’s predictions. This step is an identical to autoregressive EAGLE.

Step 2: P-EAGLE Drafter. The drafter constructs inputs for every place in parallel. Every enter consists of a token embedding concatenated with a hidden state.

For immediate positions, the enter pairs every immediate token embedding emb(p) with its corresponding h_prompt from the goal mannequin. Following the identical conference as autoregressive EAGLE, positions are shifted by one. Place i receives the token and hidden state from place i-1, enabling it to foretell the token at place i.

For place 1, Subsequent-Token-Prediction (NTP), the enter pairs the newly generated token embedding emb(new) with h_context. This place operates identically to the usual autoregressive EAGLE.For positions 2 by way of Ok, Multi-Token-Prediction (MTP), the required inputs—the token embedding and hidden state—don’t but exist. P-EAGLE fills these with two learnable parameters: a shared masks token embedding emb(masks) and a shared hidden state h_shared. These are fastened vectors realized throughout coaching that function impartial placeholders.

Positions go collectively by way of N transformer layers, then by way of the language mannequin head to foretell draft tokens t1, t2, t3, and t4 in a single ahead go.

Determine 2: P-EAGLE structure overview.

Coaching P-EAGLE on Lengthy Sequences

Fashionable reasoning fashions produce lengthy outputs. As proven in Determine 3, GPT-OSS 120B generates sequences (together with prompts) with a median size of three,891 tokens and P90 of 10,800 tokens on the UltraChat dataset. Draft fashions have to be skilled on matching context lengths to be efficient at inference.

Determine 3: Sequence size (immediate + era) distribution on UltraChat dataset with GPT-OSS 120B. Reasoning stage: Medium.

A key problem is that parallel drafting amplifies reminiscence necessities throughout coaching. Coaching Ok parallel teams on a sequence of size N creates N × Ok whole positions. With N = 8,192 and Ok = 8, a single coaching instance incorporates 65,536 positions. Consideration requires every place to attend to each legitimate place—65K × 65K means over 4 billion parts, consuming 8GB in bf16.

Place sampling [An et al., 2025] reduces reminiscence by randomly skipping positions, however skipping too aggressively degrades draft high quality. Gradient accumulation is the usual resolution for memory-constrained coaching, but it surely splits throughout completely different coaching examples. When a single sequence exceeds reminiscence, there’s nothing to separate.

P-EAGLE introduces a sequence partition algorithm for intra-sequence splitting. The algorithm divides the N × Ok place sequence into contiguous chunks, maintains appropriate consideration dependencies throughout chunk boundaries, and accumulates gradients throughout chunks of the identical sequence. For particulars, see the P-EAGLE paper.

Implementation in vLLM

Parallel drafting challenges

In lots of speculative decoding setups, drafting and verification share the identical per-request token format. That’s principally true for EAGLE: the drafter consumes a window that already matches what the verifier will test; Ok drafted tokens and one further sampled token.

Parallel drafting breaks that consistency. To foretell Ok tokens in a single drafter ahead go, we append MASK placeholders (for instance, [token, MASK, MASK, …]). These further positions exist just for drafting, so the draft batch form not matches the verification batch form. As a result of we are able to’t reuse verification metadata, we should rebuild the batch metadata. We develop the enter token IDs, hidden states, and positions to insert slots for masks tokens/embeddings, increment positions per request, then recompute the slot mapping and per-request begin indices from the up to date positions.

The Triton Kernel

To offset the overhead of rebuilding the batch metadata, we implement a fused Triton kernel that populates the drafter’s enter batch on-GPU by copying and increasing the target-model batch. In a single go, the kernel copies the earlier token IDs and positions from the goal batch into new vacation spot slots and inserts the per-request bonus token sampled by the goal mannequin. It then fills the additional parallel-drafting slots with a particular MASK token ID. Lastly, it generates light-weight metadata: a rejected-token masks, a masked-token masks for parallel drafting slots, new-token indices for sampling draft tokens, and a hidden-state mapping.

This logic would in any other case be many GPU ops (copy/scatter + insert + fill + masks + remap). Fusing it into one kernel reduces launch overhead and further reminiscence site visitors, holding the drafting setup low cost.

Hidden State Administration

For EAGLE-based strategies that go hidden states to the draft mannequin, parallel drafting handles populating these fields individually. Since hidden states are considerably bigger than the remainder of the enter batch, we cut up the work: the Triton kernel outputs a mapping, and a devoted copy kernel broadcasts the realized hidden state placeholder into the masks token slots.

# Copy goal hidden states to their new positions
 self.hidden_states[out_hidden_state_mapping] = target_hidden_states
 
# Fill masked positions with the realized Parallel Drafting hidden state
 masks = self.is_masked_token_mask[:total_num_output_tokens]
 torch.the place(
     masks.unsqueeze(1),
     self.parallel_drafting_hidden_state_tensor,
     self.hidden_states[:total_num_output_tokens],
     out=self.hidden_states[:total_num_output_tokens],
 )

The parallel_drafting_hidden_state_tensor is loaded from the mannequin’s mask_hidden buffer, a realized illustration that tells the mannequin these positions ought to predict future tokens.

For KV cache slot mapping, legitimate tokens obtain regular slot project whereas rejected tokens are mapped to PADDING_SLOT_ID (-1) to forestall spurious cache writes. For CUDA graphs, we lengthen the seize vary by Ok × max_num_seqs to accommodate the bigger draft batch launched by parallel drafting.

vLLM Benchmarking on P-EAGLE

We prepare P-EAGLE on GPT-OSS-20B and consider throughout three benchmarks: MT-Bench for multi-turn instruction following, SPEED-Bench Code for long-term code era, and HumanEval for function-level code synthesis. P-EAGLE delivers 55–69% increased throughput at low concurrency (c=1), with positive aspects of 5–25% sustained at excessive concurrency (c=64), in comparison with the publicly obtainable vanilla EAGLE-3 checkpoint. Outcomes are proven in Determine 4-6.

The P-EAGLE drafter is a light-weight 4-layer mannequin skilled to foretell as much as 10 tokens in parallel. To guage efficiency, we sweep hypothesis depths Ok ∈ {3,5,7} throughout concurrency ranges C ∈ {1,2,4,8,16,32,64}. Our aim is to determine the appropriate deployment configuration for each P-EAGLE and vanilla EAGLE-3. Linear drafting is used for each P-EAGLE and vanilla EAGLE-3. On this context, “finest P-EAGLE” and “finest EAGLE-3” confer with the configurations that obtain peak throughput. These are measured in tokens per second (TPS), for a given hypothesis depth Ok. For every technique, we choose Ok that maximizes TPS underneath the given serving circumstances.

A constant sample emerges. P-EAGLE achieves peak TPS at Ok=7 throughout all concurrency ranges. In distinction, vanilla EAGLE-3 reaches its highest TPS at Ok=3, with its improved depth sometimes shifting towards increased values relying on concurrency. This habits displays a elementary benefit of parallel drafting. P-EAGLE generates all Ok draft tokens in a single ahead go, permitting it to profit from deeper hypothesis with out incurring further sequential overhead. Autoregressive drafters, in contrast, should generate speculative tokens step-by-step, which limits their capability to effectively scale to bigger Ok.

All experiments are performed on one NVIDIA B200 (Blackwell) GPU utilizing vLLM with the next serving configuration.

VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8=1 
vllm serve openai/gpt-oss-20b 
    --speculative-config '{
      "technique": "eagle3",
      "mannequin": "amazon/GPT-OSS-20B-P-EAGLE",
      "num_speculative_tokens": 7,
      "parallel_drafting": true}' 
    --port 8000 
    --max-num-seqs 1024 
    --max-model-len 100000 
    --max-num-batched-tokens 100000 
    --max-cudagraph-capture-size 4096 
    --no-enable-prefix-caching 
    --no-enable-chunked-prefill 
    --kv-cache-dtype fp8 
    --async-scheduling 
    --stream-interval 20

Word. Serving GPT-OSS-20B with EAGLE drafters at the moment requires a one-line vLLM patch (PR#36684). Apply it earlier than launching. This repair is anticipated to land in an upcoming vLLM launch.

Determine 4: MT-Bench throughput (TPS) for P-EAGLE vs EAGLE-3 on GPT-OSS-20B throughout concurrency ranges. The P/E speedup ratios are: 1.55x (c=1), 1.29x (c=2), 1.35x (c=4), 1.28x (c=8), 1.27x (c=16), 1.09x (c=32), and 1.05x (c=64).

Determine 5: HumanEval throughput (TPS) for P-EAGLE vs EAGLE-3 on GPT-OSS-20B throughout concurrency ranges. The P/E speedup ratios are: 1.55x (c=1), 1.53x (c=2), 1.45x (c=4), 1.35x (c=8), 1.31x (c=16), 1.37x (c=32), and 1.23x (c=64).

Determine 6: Pace-bench throughput (TPS) for P-EAGLE vs EAGLE-3 on GPT-OSS-20B throughout concurrency ranges. The P/E speedup ratios are: 1.69x (c=1), 1.61x (c=2), 1.54x (c=4), 1.45x (c=8), 1.40x (c=16), 1.22x (c=32), and 1.25x (c=64).

Along with lowering drafting overhead, P-EAGLE’s throughput positive aspects are additionally pushed by higher acceptance size (AL), the common variety of draft tokens accepted by the verifier per hypothesis spherical. Larger AL means extra of the draft work turns into actual output, which immediately boosts efficient OTPS/TPS.

The next tables evaluate AL for P-EAGLE and vanilla EAGLE-3 on GPT-OSS-20B throughout our three benchmarks:

P-EAGLE (AL):

EAGLE-3 (AL):

P-EAGLE persistently achieves increased AL than EAGLE-3 on the identical hypothesis depth Ok. At Ok=7, P-EAGLE outperforms EAGLE-3 by 30% on HumanEval (3.94 vs 3.03), 31% on SPEED-Bench (3.38 vs 2.59), and 13% on MT-Bench (3.70 vs 3.27).Notably, P-EAGLE advantages extra from deeper hypothesis. From Ok=3 to Ok=7, P-EAGLE’s AL will increase by 0.92 on HumanEval (3.02 to three.94), whereas EAGLE-3 positive aspects solely 0.38 (2.65 to three.03). This widening hole at increased Ok is per P-EAGLE’s single-pass parallel drafting, which incurs no further value from deeper hypothesis.

Reproducing the Outcomes

After launching the server, run benchmarks with `vllm bench serve`:

#MT-Bench                                                                           
export MODEL="openai/gpt-oss-20b"
export BASE_URL="http://localhost:8000" 
vllm bench serve 
    --dataset-name hf 
    --dataset-path philschmid/mt-bench 
    --num-prompts 80 
    --max-concurrency 1 
    --model $MODEL 
    --base-url $BASE_URL 
    --temperature 0.0 
    --hf-output-len 2048
 
#HumanEvalcommand: 
#Obtain HumanEval dataset openai/openai_humaneval                            
vllm bench serve 
    --dataset-name {custom}                                                             
    --dataset-path  
    --num-prompts 164 
    --max-concurrency 1 
    --model $MODEL 
    --base-url $BASE_URL 
    --temperature 0.0 
    --custom-output-len 2048

P-EAGLE removes the sequential bottleneck from speculative decoding, delivering as much as 1.69× speedup over vanilla EAGLE-3 on actual workloads. By decoupling draft rely from ahead go rely, we are able to now discover bigger drafting architectures, which may even allow elevated acceptance charges in comparison with single-layer baselines. This implementation rigorously handles the complexities of enter preparation, consideration metadata administration, and KV cache slot mapping by way of hand-written fused kernels. Whereas it requires specifically skilled fashions, the efficiency advantages make it a invaluable addition to vLLM’s speculative decoding capabilities.

As extra parallel-trained fashions grow to be obtainable, we anticipate this method to grow to be the popular selection for manufacturing LLM deployments. The mix of P-EAGLE’s architectural effectivity and vLLM’s sturdy infrastructure gives a transparent path for these searching for most inference efficiency and diminished latency.

Attempt it right this moment: obtain a pre-trained P-EAGLE head from HuggingFace, set "parallel_drafting": true in your vLLM config for any of the supported fashions, and see the speedup for your self.

Acknowledgement

We wish to acknowledge our contributors and collaborators from Nvidia: Xin Li, Kaihang Jiang, Omri Almog, and our workforce members: Ashish Khetan, and George Karypis.

Concerning the authors