Any engineer currently running Eagle3 or a multi-token-prediction setup to speed up their LLM inference now has a compelling reason to look at the alternative NVIDIA’s research team published Tuesday. The model is free, the weights are live on Hugging Face, and it eliminates the separate draft model entirely.
NVIDIA researchers NVIDIA’s July 2026 arXiv preprint, introducing Nemotron-Labs-Diffusion, a language model family that operates in three distinct generation modes from a single set of weights. The models — available in 3B, 8B, and 14B parameter sizes — are downloadable now from the NVIDIA collection on Hugging Face under a license that permits commercial use.
The core result: the 8B instruct variant decodes six times more tokens per forward pass than Qwen3-8B in self-speculation mode and translates that to four times higher throughput on SPEED-Bench when served with SGLang on a GB200 GPU, according to the arXiv preprint. Those figures come from an NVIDIA-authored preprint that has not yet undergone peer review or independent replication — the standard caveat for any arXiv paper published two days ago.
Three Modes, One Checkpoint
Every large language model in widespread deployment today generates text autoregressively: one token per forward pass, left to right, each new token requiring the full weight matrix to be loaded from GPU memory before computation can proceed. At high concurrency — hundreds of users simultaneously — this sequential bottleneck matters less, because the GPU stays busy processing batches. At low concurrency, the kind typical of a single developer’s API calls, an agentic loop, or an edge deployment, the GPU sits largely idle between each token step. Hardware benchmarks confirm that single-token LLM inference runs deep in the memory-bandwidth-bound regime, roughly two orders of magnitude below the compute-bound ridge point, meaning GPU arithmetic units are waiting on data movement, not running out of math to do.
Diffusion language models address this by replacing one-token-at-a-time commitment with block-level parallel denoising. Rather than emitting tokens in order, a diffusion model starts from a block of masked positions and refines all of them simultaneously across several passes, committing high-confidence positions at each step. The approach has progressed rapidly — Google DeepMind’s Gemini Diffusion demonstrated commercial-grade quality parity with autoregressive models in May 2025 — but diffusion models have historically required substantially more training data to match autoregressive quality because they treat all token orderings as equally likely, ignoring the strong left-to-right structure that natural language actually exhibits.
Nemotron-Labs-Diffusion’s claim is that these two paradigms do not need to compete. Trained on a joint autoregressive-diffusion objective, it can switch among three modes at inference time by changing only the attention pattern — no weight updates, no separate file:
Autoregressive mode uses standard causal attention, generating one token per forward pass. The paper recommends it for high-concurrency cloud serving, where a large batch of simultaneous requests keeps the GPU saturated regardless of per-token latency.
Diffusion mode processes a fixed-length block of positions simultaneously. Within each block, tokens attend bidirectionally; across blocks, attention remains causal, so prior blocks can reuse their key-value caches. A lightweight trained confidence sampler predicts, per masked position, whether the model’s top prediction at the current denoising step is likely correct. Positions crossing the threshold are committed in that step, allowing multiple tokens to be finalized per forward pass. The threshold is adjustable — higher values trade throughput for accuracy.
Self-speculation mode is the paper’s most novel contribution, and the mode where the largest throughput gains appear.
Self-Speculation Eliminates the Draft Model Problem
Speculative decoding research has been a production inference technique since Google demonstrated it in 2022 and it entered mainstream serving frameworks through 2025. The technique’s central idea is to run a smaller “draft” model to propose candidate tokens and a larger “target” model to verify them in parallel. Because verifying multiple tokens in one forward pass is roughly as expensive as generating a single token, the technique can deliver 2–3× speedup without any change in output quality.
The practical limitation is the draft model itself. Eagle3, currently the highest-quality auxiliary-head method for general-purpose LLMs, published at NeurIPS 2025, attaches a small auxiliary draft component trained on the target model’s hidden states. It is highly effective — delivering 3.0–6.5× speedup versus vanilla autoregressive decoding — but it requires a separate component that must be maintained alongside the main model, loaded into GPU memory simultaneously, and re-tuned when the base model changes. For a team operating under VRAM constraints — a single-GPU inference instance, an edge device, or a deployment environment where only one model checkpoint fits in memory — this dual-model requirement is a structural barrier.
Self-speculation in Nemotron-Labs-Diffusion collapses that requirement entirely. The same weights serve as both drafter and verifier. In self-speculation mode, the diffusion attention pathway generates a block of candidate tokens in parallel; the autoregressive attention pathway then runs a second forward pass over those candidates and accepts the longest contiguous prefix that matches its own predictions. Both roles — drafting and verification — are filled by the same weight checkpoint, with a shared key-value cache. This means the total operational overhead of self-speculation is essentially the overhead of running one model, not two.
A small LoRA adapter — approximately 36 million parameters, about 0.4 percent of the 8B backbone — can optionally be applied to the diffusion draft pathway to better align its outputs with the autoregressive verifier, yielding further gains in accepted tokens per step with negligible accuracy change, according to the paper. Without the adapter, self-speculation already outperforms competing multi-token prediction methods; with it, the gap widens.
How the Training Makes It Work
The paper’s central finding is that training simultaneously on autoregressive and diffusion objectives is not a zero-sum tradeoff. The researchers found that diffusion training improves “lookahead planning” — the model’s ability to anticipate what should come several positions ahead — while autoregressive training provides the left-to-right linguistic priors that make language generation coherent. Both modes peaked at the same weighting coefficient (α = 0.3 for the diffusion loss) and across a range of values from 0.1 to 0.5, neither mode improved at the other’s expense.
Training proceeded in two stages. The model first trained purely on the autoregressive objective for approximately one trillion tokens, building strong left-to-right priors on a Ministral3 base model. It then continued on the joint objective for an additional 300 billion tokens across 256 NVIDIA H100 GPUs. Ablation experiments credited the two-stage approach with a 5.74 percent average accuracy gain on top of other contributions; adding the autoregressive loss itself provided the single largest individual improvement at 7.48 percent. The cumulative pipeline improvement over the baseline reached 16.05 percent.
Performance Against Current Speculative Decoding Methods
On a ten-task evaluation covering coding (HumanEval, MBPP, LiveCodeBench), math (GSM8K, Math500, AIME24, AIME25), scientific reasoning (GPQA), instruction-following (IFEval), and general knowledge (MMLU), detailed in the paper, the 8B instruct model in autoregressive mode scored 63.61 percent average, compared to 62.75 percent for Qwen3-8B and 58.02 percent for Ministral3-8B-Instruct. Self-speculation with the LoRA adapter delivered 5.99× tokens per forward pass at 62.81 percent average accuracy — a fraction lower, but within the margin that most production teams would accept for a 6× throughput improvement.
The comparison against Eagle3 is the figure most relevant to teams currently running speculative decoding in production. Across SPEED-Bench categories, Nemotron-Labs-Diffusion with the LoRA adapter averaged 6.82 accepted tokens per draft step. Eagle3 averaged 2.75. Qwen3-9B-MTP, the multi-token prediction baseline, averaged 4.24. On structured-content categories — coding, math, reasoning, and multilingual tasks — the gap widened further: Nemotron-Labs-Diffusion averaged 8.69 accepted tokens per step versus 2.81 for Eagle3.
At the 14B scale, the model achieved 66.36 percent average accuracy at 5.96× tokens per forward pass, outperforming Qwen3-14B’s 65.17 percent in standard autoregressive mode. The vision-language variant extended the same framework to multimodal tasks, achieving 3.63× to 7.45× tokens per forward pass in self-speculation mode — the upper range occurring for responses longer than 200 tokens — with only a 0.1 percent average accuracy drop versus autoregressive mode.
How Much Faster Self-Speculation Can Get
One of the paper’s more informative contributions is a “speed-of-light” analysis — an estimate of the theoretical upper bound on parallel decoding, assuming an oracle sampler that correctly identifies every position that can safely be committed in parallel, described in the paper.
At a block length of 32, an oracle sampler would achieve 7.60× tokens per forward pass on average and exceed 10× on coding and multilingual content. The current confidence-based sampler achieves approximately 3× tokens per forward pass at comparable accuracy. The gap between today’s practice and the theoretical ceiling is large, and the researchers note that designing samplers that approach this ceiling remains an open problem. Practical translation: the 6.82 accepted-tokens-per-step figure, impressive against today’s competitors, likely understates what further sampler improvements alone could deliver from the same underlying architecture.
This ceiling gap is the most important long-range signal in the paper. The architecture has been demonstrated; the training recipe is public; the weights are available. What comes next — from NVIDIA, from the research community, and from engineers adapting the approach to other model families — is a sampler design problem, and the oracle analysis defines exactly how much headroom remains.
Switching Modes at Inference Time
Mode switching requires no architectural modification, only a different API call on the same model object. In autoregressive mode, the standard call is ar_generate(). In diffusion mode, generate() with configurable block length and confidence threshold. In self-speculation mode, linear_spec_generate() or, for the quadratic variant that prepares verification positions simultaneously, a separate method that reaches 6.38× tokens per forward pass and 64.04 percent average accuracy on the 8B model.
The model requires trust_remote_code=True on load because custom modeling code is bundled with the Hugging Face checkpoint. CUDA-capable GPU hardware is required. Optional PEFT installation enables the LoRA-enhanced self-speculation mode. Standard inference stacks — vLLM and SGLang — both support the model.
The paper’s authors describe the design as a system that offers “consistently high throughput across deployment scenarios” without architectural or pipeline changes. The weights are available at nvidia/Nemotron-Labs-Diffusion-3B, -8B, and -14B on Hugging Face.
Frequently Asked Questions
What is self-speculation in an LLM, and why does it matter for inference?
Self-speculation is a variation of speculative decoding in which the same model serves as both the draft generator and the verifier, eliminating the separate draft model that conventional speculative decoding requires. In Nemotron-Labs-Diffusion, the diffusion attention pathway proposes candidate tokens for a block in parallel, and the autoregressive attention pathway verifies them — all within the same weight checkpoint and shared key-value cache. This matters because it makes high-quality speculative decoding available without the operational overhead of loading and maintaining two models simultaneously, which has historically limited speculative decoding’s adoption in single-GPU or VRAM-constrained deployments.
How does this compare to Eagle3, and should I switch?
Eagle3, currently the leading auxiliary-head speculative decoding method, averages 2.75 accepted tokens per draft step on SPEED-Bench. Nemotron-Labs-Diffusion with the LoRA adapter averages 6.82 — a 2.5× gap on that acceptance-length metric. Eagle3 requires a separately trained draft component keyed to your target model’s hidden states; Nemotron-Labs-Diffusion self-speculates from a single checkpoint. The tradeoff is that switching means adopting a new base model family (8B initialized from Ministral3) rather than accelerating a model you have already fine-tuned. For new deployments or teams with flexibility on model choice, the weights-available-now case for switching is meaningful. For teams with significant fine-tuning investment in a non-Nemotron model, the path is more complex. The paper is a two-day-old preprint and has not yet been independently replicated.
Is there a ceiling on how fast self-speculation can get, and how far are we from it?
Yes. The paper’s “speed-of-light” analysis shows that an oracle confidence sampler — one that perfectly identifies every position that can safely be committed in parallel — would achieve 7.60× tokens per forward pass on average at block length 32, exceeding 10× on coding and multilingual tasks. The current sampler achieves approximately 3× tokens per forward pass. The 6.82 accepted-tokens-per-step figure from the SPEED-Bench tests reflects a specific serving configuration (SGLang, GB200 GPU) rather than the raw sampler ceiling. The researchers identify designing samplers that approach the oracle ceiling as an open problem, meaning further throughput gains from sampler research alone — without changing the training procedure or the model weights — are plausible.
Can I use these models commercially, and where do I download them?
Yes. The weights are released under the NVIDIA Nemotron Open Model License, which permits commercial use. Models are available at nvidia/Nemotron-Labs-Diffusion-3B, -8B, and -14B on Hugging Face, along with base and vision-language variants. Inference is supported through vLLM and SGLang. The preprint describing the architecture and training procedure is freely available at the paper on arXiv.
