Jiajun Fan

Vanguard: Knowing When to Trust the Draft in Speculative Vision‑Language Agents

  • 1UIUC
  • 2CMU
  • 3NYU
  • 4Northeastern

*Equal contribution

At a glance

The paper in four results.

What if the small model went first, and the large model wrote only the steps it gets wrong? With a judge that checks each drafted step against the screen, the small model’s drafts can be more accurate than the large model alone.

  • Training-free.only the question and one threshold are set, on development data
  • Across model families.no draft heads and no shared tokenizer, so the target can be another family
  • Sees the screen.each drafted step is judged with the screenshot in view

The problem

Step-level speculation rests on its judge, and it has been measured by how little it loses.

Agents that act from screenshots reason before every action, so every step waits for a large model. Step-level speculation lets a small model draft the step, and the large model writes only the steps a judge rejects.

The large model grades

Its probabilities sit almost all near 0 or 1, so a threshold on them accepts or rejects nearly everything.

A process reward model

It is specialised to one domain and must be trained for it.

The small model’s own signals

They inherit whatever it fails to notice about its own mistakes.

A judge that reads only text

A fluent step can click an element that is not on the screen, and a text-only judge cannot tell.

The judge decides what is saved and what is lost, and whether the small model can end up ahead.

The finding

Four drafts of a small model can hold a right step about as often as the large model is right.

Sample the 2B draft four times on the same step: one of its four answers is right about as often as the 8B gets the step right, and not always on the same steps.

Qwen3-VL 2B draft, four samples at temperature 0.7, against the Qwen3-VL 8B target decoding greedily. Mind2Web pools held-out tasks and unseen websites.

The raw material for beating the large model is already there. What is missing is a judge that can pick it out.

How it works

One question, asked with the screen in view.

For each drafted step, a general decision model is asked whether the step is right, given the goal, the screenshot and the previous actions. The best draft is kept if the answer is confident; otherwise the large model writes the step.

  • no training
  • no draft heads
  • no shared tokenizer
  • no generated text
  1. 1Draft

    The small model samples candidate steps, each with its reasoning and an action.

    K = 4 candidates per step

  2. 2Ask

    A general decision model sees the goal, the screenshot, the previous actions and one proposed action. It does not see the candidate’s reasoning.

    One yes-or-no question per candidate

  3. 3Score

    It reads the probability of yes in one forward pass, without generating any text.

    P(yes) for each candidate

  4. 4Keep or fall back

    The best candidate is kept if its probability clears τ. Otherwise the large model writes the step itself.

    τ = 0.5 on web and phone tasks, 0.451 in homes

Training-free. Only the wording of the question and τ are chosen, on development data; the decision model is used as it is. The draft needs no heads trained for the target, so the target can come from another family, and it composes with token-level speculation.

On web and phone tasks it has to see the screen: with the screenshot removed, or swapped for another task’s, it falls to or below the large model.

The evidence Finding 01

A small draft surpasses its large target.

On Multimodal-Mind2Web the 2B draft judged by Vanguard is significantly more accurate than the 8B alone on websites it has never seen, while the 8B writes about a quarter of the steps.

An unseen website · one held-out step, the 2B draft judged by Vanguard

  1. Drafts 1 and 2click G, Intel Core i3D 0.97 ≥ τ: kept
  2. Drafts 3 and 4click H, Intel Core i7D 0.02
  3. 8B alonetypes into the memory fieldwrong

Mind2Web step success (%). Every judge gets the same four drafts and the same fallback to the 8B. *: significantly above the 8B alone (exact McNemar test), as is Vanguard on both test sets together. In the example, the decision model separates the right drafts from the wrong ones, and the step is kept.

Given the same four drafts, none of the other judges shown here surpasses the 8B significantly. Vanguard does, on websites it has never seen.

The evidence Finding 02

It matches the large model on phones, beats it in homes, and in closed loop solves far more episodes.

When each step builds on the agent’s own previous actions, better steps add up.

BabyAI GoToLocal, closed loop · one held-out episode: go to a grey key

8B alone: fails after 32 actionswalks up the right column and repeats itself
Vanguard: reaches the key in 11 actionskept draft steps up, one 8B step, then left to the key

Success (%) on AndroidControl (525 steps) and EB-ALFRED (1,849 steps); episodes solved out of 100 on the two BabyAI tasks. With each Vanguard bar: the share of steps the 8B writes. *: significant gain over the 8B alone.

In closed loop the 2B draft judged by Vanguard solves 86 episodes where the 8B alone solves 61, and 65 where it solves 36.

The evidence Finding 03

Across model families: faster and more accurate.

Vanguard exchanges whole steps, not tokens, so the target can come from another family. With a Qwen3-VL 2B draft and a Gemma-3 27B target, it is faster than the 27B alone and significantly more accurate on all three agent tasks.

Step success (%) on the full test sets; all seven gains are significant. Speed: end to end at batch size 1 against the 27B alone, on the first 300, 200 and 300 test steps of Mind2Web, AndroidControl and EB-ALFRED.

No draft heads, no shared tokenizer: a small model of one family can lead a large model of another.

The evidence Finding 04

One rule says when it pays off.

A judge can only choose among the drafts, so the small model can surpass its target only where its four drafts hold a right step that the target misses. Across sixteen results in four model families, the gain grows with the headroom: how often one of the four drafts is right, minus how often the target is right.

Sixteen (draft, target, task) results across Qwen3-VL, Qwen2.5-VL, Qwen3.5 and Gemma-3, on Mind2Web, AndroidControl and EB-ALFRED. Filled: significant gain over the target. Dashed: least-squares line. Pearson r = 0.95.

Where the headroom is largest, Vanguard turns more than half of it into a significant gain.

Why it matters

A judge that can tell lets the large model write only the steps the small one gets wrong.

Vanguard is the most accurate judge compared on the agent tasks, tied only by Qwen3-VL-32B on Mind2Web, and its scores are calibrated, so one threshold carries over from development data.

Judge accuracy over three model families and three agent tasks. With one τ = 0.5, the 8B writes 23% of the development steps, 26% on held-out tasks and 23% on unseen websites.

What this opens up

Nothing to train

No draft heads, no shared tokenizer, no new reward model. The draft and the target stay as they are.

With token-level speculation

With EAGLE-3 inside both models, Vanguard comes close to EAGLE-3’s speed and, on unseen websites, is significantly more accurate than EAGLE-3 alone: 68.4% against 63.5%.

Beyond agents

On MathVista and ProcessBench it is on par with process reward models trained for these domains; picking one of four solutions on MATH-500, it significantly beats the math reward model.

Speculation was measured by how little it loses. Now the small model can lead.

The film

The story, as a short film.

2 min 54 s, narrated. Also in the film collection.

Cite

Cite the paper.

@article{mao2026vanguard,
  title={Vanguard: Knowing When to Trust the Draft in Speculative Vision-Language Agents},
  author={Mao, Hanqi and Fan, Jiajun and Guo, Ruihan and Shen, Shuaike and Wang, Ziwen and Ren, Hui and Zhong, Xianrui and Liu, Ge},
  journal={arXiv preprint arXiv:2611.23917},
  year={2026},
  url={https://arxiv.org/abs/2611.23917}
}