Voice Agents Could Be Ranked, Never Improved. We Closed the Loop.
A voice agent has to do everything a text agent does — call tools against a live database, respect a policy, drive a conversation to a verifiable end state — with speech as its only channel. The standard recipe trains the policy in text and bolts speech on at the edges. This note is about what happens when you stop doing that, and train the agent in audio end to end.
The loop, closed and local. The caller is not a recording and not a text simulator wired to a speech engine — it is an open model generating its turns as audio, inside the same process as the agent. That single choice is what turns an evaluation harness into a gym: gradients reach the policy, a rollout costs compute instead of API credit, and the reward is whatever the underlying benchmark already checks. Tasks, tools, database and success check are inherited unmodified, so the channel is the only variable.
The short version
Four things a closed loop made visible
The agent picks the right tool and the right argument slot, then fills it with a value it mis-heard from the waveform. That happens in 32% of speech rollouts against 2% in text — a sixteen-fold gap, and the single largest difference between the channels.
A perceptual error labels itself: a call with a mis-heard argument fails against the database while a correct one succeeds. The environment supplies that signal for free — no annotation, no reward model.
Near the floor almost every group of rollouts fails identically, so a group-relative objective sees no variance and produces no gradient: only 16% of groups carried one. Crediting each successful tool call restores that to 99.6%.
Run untuned inside an independently implemented benchmark — different user simulator, different TTS and ASR, different grader — pass@1 more than doubles, 24% → 53%, and an open 30B model moves from last place to second.
01 — What breaks
It hears the wrong digit, and the episode is already lost
Run the same tasks, the same tools and the same reward down two channels, changing nothing but whether the caller types or speaks. Every failure mode gets worse in speech, but not evenly — and the one that moves most is not a reasoning failure at all.
Same tasks, same tools, same reward — only the channel differs. Mis-hearing is near-absent in text and ordinary in speech; the errors downstream of it are merely more common. That ordering is the whole diagnosis: the agent understands the task and picks the right call, then writes a digit of a zip code, a character of an order ID or a spelling of a name that it never heard correctly.
Two of these are not separate problems but one cascade. A mis-heard value produces a wrong argument; the wrong argument produces a tool error; the error produces a retry of the identical call; and the retries burn the step budget until the episode times out at zero reward. An outcome-only view sees one undifferentiated failure. The environment sees four distinct events, each with its own signal.
A third failure has no counterpart in the text baseline: under an insistent caller the agent performs a state-changing write it was not authorised to perform, and ends the episode believing it helped. Pressure and urgency live in a voice and are flattened by a transcript. That one is reported qualitatively — there is no matched text number for it.
02 — The obstacle
Not optimisation. Starvation.
Given a trainable loop and a labelled failure, the fix looks like ordinary RL. It is not, and the reason is arithmetic rather than audio. A group-relative objective normalises returns within a group of rollouts on the same task. When almost every episode fails, every rollout in the group returns zero, the standard deviation is zero, and the group contributes nothing — its compute is spent generating trajectories that never touch the weights.
Each square is one group of rollouts. Under outcome-only reward, 84 of every 100 groups are generated and then discarded — four fifths of the rollout budget producing no gradient at all. Crediting each successful tool call as it happens lets two rollouts that both failed still differ in how far they got, and almost every group becomes informative.
The shaping is deliberately unremarkable: +0.1 for each successful tool execution, −0.1 for each failed one, with the episode outcome appended to the final step so the terminal criterion is untouched. Four failing rollouts that scored {0, 0, 0, 0} under outcome-only reward might now score {0.30, −0.10, 0.20, 0.10} — still all failures, but now ordered, and an ordering is a gradient.
Note what this does not change. The reward that decides whether an episode succeeded is still the benchmark’s own check, run unmodified. The shaping only decides which rollouts are worth learning from along the way.
03 — It transfers
Trained in our environment, tested in someone else’s
The obvious worry about any RL result is that the policy learned the environment rather than the task. So the trained checkpoint was run, with no further tuning, inside a separate evaluation-only voice benchmark built by a different group: its own cascaded user simulator, its own TTS voices over a telephony-grade channel, its own grader. Acoustics, user policy and scoring all change at once. Only the agent’s weights are ours.
Every pair comes from the same external pipeline. Same tasks, same user simulator, same acoustics, same grader, single attempt — only the weights differ. The gain holds in all three domains and the 95% confidence intervals do not overlap in any of them. The largest relative move is Telecom, the dual-control domain where the user also holds tools: a six-fold rise from a base of 4%.
Where that leaves an open model. The same 30B open-weights model moves from last place to second on this benchmark. The other systems are commercial stacks we did not build, train or re-run — their scores are as reported by their own providers, so read this as context rather than a controlled comparison. The controlled part is the pair at the bottom and the middle: identical model, identical benchmark, RL in between.
04 — Did it game the metric?
Success rose while the agent got cheaper
A jump from 24% to 53% invites the suspicion that a number was gamed rather than a task solved. Two checks, either of which could have falsified it.
Did the diagnosed failures actually go away?
Had the gain come from somewhere other than the cascade of §01, the rates that cascade produces would be roughly unchanged. They are not: every behaviour the environment was built to expose moved in the intended direction, measured on the raw trajectories of both models over the same external task set.
Did it buy the win with more talking?
Inflating a success rate by spending more of the episode — retrying, or stalling until the user concedes — predicts turns and tokens rising alongside pass@1. They fall.
On the tasks the trained model actually fixes it is 25% more concise than the base was. Success rising while compute falls is the opposite of length-based reward hacking.
The annotations also record repair strategies nobody designed: asking the caller to spell a name out, retrying a lookup with a corrected spelling, switching the lookup key when the first one fails. Nothing in the reward mentions spelling, retries or lookup keys — it scores task completion and tool-call success. RL found them because they raise the chance of finishing a task over an unreliable channel.
05 — What it costs to run
The loop has to be cheap enough to spin
An episode here is dozens of turns, each generating audio from two 30B models. Rollout collection, not the gradient step, is the bottleneck — so serving those rollouts efficiently is what makes online speech RL practical at all.
Seven epochs. A run that costs over $1,400 in API calls costs nothing here, and the money was never the real obstacle — the rollout volume on-policy RL needs simply cannot be bought at that price.
Why the loop has to be local. On-policy RL needs to sample from the current policy, over and over. Behind a commercial voice API the gradients cannot cross the boundary at all, and even ignoring that, the per-rollout price rules out the volume: the paper puts one epoch at over $200, a seven-epoch run over $1,400, and a thousand-epoch budget near $200k. With both models open and local the same loop costs compute and nothing else.
Where the time goes. Batching the rollouts through an inference server cuts a full epoch from 15 hours to 2.8. The policy update is untouched at ten minutes — it was never the bottleneck. Multi-turn speech rollouts were, and they are the part that a commercial API also makes expensive.
06 — Scope
What this does not show
The user simulator is a model, not a person: it fixes the speech distribution that policy improvement is measured against, and an idealised one. Self-play audio is clean; the external benchmark supplies the noisier test, which is why the headline result is the one measured outside the training environment. Confidently-wrong over-action is reported qualitatively against text because there is no matched text baseline for it. Results are one model family on both sides of the conversation. And the commercial systems in the standing were evaluated by their own providers on their own deployed stacks — we did not build, train or re-run them.
Why it matters
The text-to-audio gap is a deficit, not a tax
The gap between what an agent can do in text and what it can do in speech has been reported as a property of the modality — a cost you pay for using voice. Closing the loop recasts it as something with an objective, a gradient and a stopping criterion: identifiable competences — hearing a slot value correctly, confirming before acting, abandoning a failing plan — that closed-loop training substantially reduces. And because the interface is an ordinary reset/step/reward loop over text agentic domains, any open omni-modal model drops into it.