TL;DR — Voice-agent benchmarks score a proprietary realtime API from the outside: no gradient crosses the boundary, and per-call pricing rules out the rollout volume RL needs. SpeechGym makes both sides of the conversation local open models talking in native speech over an existing tool-use benchmark, so a voice agent can actually be trained. A per-turn process reward then fixes the thing that breaks first: at audio success rates, almost every rollout group fails identically and teaches nothing.
A cascade of ASR, a text user simulator and TTS plays the caller to a closed realtime API. You can score it; you cannot differentiate through it, and one epoch of rollouts costs real money.
SpeechGym
A frozen open omni-model plays the caller in native speech; the same architecture, trainable, plays the agent. Tasks, tools, databases and the success check come from the text benchmark unchanged — only the channel becomes audio.
Because both models run locally there is no API bill, and a batched rollout server makes an epoch 5.4× faster, which is what makes on-policy RL affordable at all.
Both sides run locally: the success check becomes a training signal, and rollouts cost compute, not API calls.
How to read it. In evaluation-only frameworks a cascade of ASR, a text user simulator and TTS plays the caller to a closed realtime API: each call is billed, the benchmark gets a score, and nothing can be differentiated back through the API. SpeechGym puts open models on both sides of the call and keeps the text benchmark's tasks, tools and success check unchanged, so that success check now trains the agent. Schematic.
What speech actually breaks
Running the same tasks over audio instead of text is a controlled experiment, and the failures are perceptual rather than reasoning failures:
Table 1. How often each failure happens on the same tasks over speech and over text.
Failure mode
Speech
Text
Speech ÷ text
Mis-hearing a slot value (name, ID, digit)
32%
2%
16×
Tool-call error rate
42%
26%
1.6×
Dead loop, retrying the same failing call
29%
18%
1.6×
These are not independent problems but one cascade: a mis-heard argument produces a tool error, the error produces a retry, and the retry burns the step budget into a terminal zero. Every link in it already has a free reward channel — which is the opening.
One cascade, not three problems — and every link already has a free reward channel.
How to read it. Over audio the failures are perceptual, and they chain: a mis-heard argument produces a tool error, the error produces a retry, and the retries use up the step budget before the task can finish. The bars under each stage are speech versus text on the same tasks, drawn to scale; the ID is illustrative. Every link already has a free reward channel — the opening for the per-turn reward below. Schematic.
A per-turn reward keeps the rollouts informative
GRPO learns from differences inside a group. At audio base success rates nearly every group of rollouts fails identically, the advantage is zero, and the compute is wasted. Scoring each turn — a small bonus for a tool call that executes, a penalty for one that fails, the benchmark's own outcome still appended at the end — separates rollouts that got further from rollouts that did not.
How to read it. Both sides of the call are local open models talking in speech, so the agent can be trained. When every rollout in a group fails the task, outcome-only GRPO sees identical rewards and learns nothing; scoring each turn (+ for a tool call that executes, − for one that fails) still tells the rollouts apart. Schematic.Figure 1. Share of rollout groups that still carry a gradient. Outcome-only training discards 84% of its groups; the per-turn process reward keeps almost all of them.
It transfers out of the gym
The trained checkpoint is run, with no further tuning, inside τ-Voice — an independently built benchmark with its own user simulator, its own acoustics and its own grader. Task success goes from 24% to 53%, improving in every domain, while the agent uses fewer turns and tokens than before training.
Figure 2. Training moves the same open 30B model from last place to second on τ-Voice. Other systems are as reported by their providers; we did not train or re-run them.
Limitation worth stating: the caller is a simulated model, not a person. The transfer result is measured outside the training environment precisely because in-gym numbers would be self-flattering.
BibTeX
@article{fan2026speechgym,
title = {SpeechGym: An Audio-Native Gym for Training Voice Agents
via Reinforcement Learning},
author = {Jiajun Fan and Jingyuan Li and Prashanth Gurunath Shivakumar
and Jia-Hong Huang and Qi Luo and M. Maruf and Ivan Bulyko
and Ge Liu and Roger Ren},
journal = {arXiv preprint arXiv:2608.26432},
year = {2026}
}