SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning

TL;DR — Voice-agent benchmarks score a proprietary realtime API from the outside: no gradient crosses the boundary, and per-call pricing rules out the rollout volume RL needs. SpeechGym makes both sides of the conversation local open models talking in native speech over an existing tool-use benchmark, so a voice agent can actually be trained. A per-turn process reward then fixes the thing that breaks first: at audio success rates, almost every rollout group fails identically and teaches nothing.

Why voice agents could not be trained

Evaluation-only frameworks

A cascade of ASR, a text agent and TTS around a closed realtime API. You can score it; you cannot differentiate through it, and one epoch of rollouts costs real money.

SpeechGym

A frozen open omni-model plays the caller in native speech; the same architecture, trainable, plays the agent. Tasks, tools, databases and the success check come from the text benchmark unchanged — only the channel becomes audio.

Because everything runs locally, an epoch is 5.4× faster at zero API cost, which is what makes on-policy RL affordable at all.

What speech actually breaks

Running the same tasks over audio instead of text is a controlled experiment, and the failures are perceptual rather than reasoning failures:

Failure mode	Speech	Text
Mis-hearing a slot value (name, ID, digit)	32%	2%	16×
Tool-call error rate	42%	26%	1.6×
Dead loop, retrying the same failing call	29%	18%	1.6×

These are not independent problems but one cascade: a mis-heard argument produces a tool error, the error produces a retry, and the retry burns the step budget into a terminal zero. Every link in it already has a free reward channel — which is the opening.

A per-turn reward keeps the rollouts informative

GRPO learns from differences inside a group. At audio base success rates nearly every group of rollouts fails identically, the advantage is zero, and the compute is wasted. Scoring each turn — a small bonus for a tool call that executes, a penalty for one that fails, the benchmark's own outcome still appended at the end — separates rollouts that got further from rollouts that did not.

Figure 1. Share of rollout groups that still carry a gradient. Outcome-only training discards 84% of its groups; the per-turn process reward keeps almost all of them.

It transfers out of the gym

The trained checkpoint is run, with no further tuning, inside τ-Voice — an independently built benchmark with its own user simulator, its own acoustics and its own grader. Task success goes from 24% to 53%, improving in every domain, while the agent uses fewer turns and tokens than before training.

Figure 2. Training moves the same open 30B model from last place to second on τ-Voice. Other systems are as reported by their providers; we did not train or re-run them.

Limitation worth stating: the caller is a simulated model, not a person. The transfer result is measured outside the training environment precisely because in-gym numbers would be self-flattering.

BibTeX

@article{fan2026speechgym,
  title   = {SpeechGym: An Audio-Native Gym for Training Voice Agents
             via Reinforcement Learning},
  author  = {Jiajun Fan and Jingyuan Li and Prashanth Gurunath Shivakumar
             and Jia-Hong Huang and Qi Luo and M. Maruf and Ivan Bulyko
             and Ge Liu and Roger Ren},
  journal = {arXiv preprint arXiv:2608.26432},
  year    = {2026}
}