SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning
Published in arXiv preprint, 2026, 2026
Recommended citation: Jiajun Fan, Jingyuan Li, Prashanth Gurunath Shivakumar, Jia-Hong Huang, Qi Luo, M. Maruf, Ivan Bulyko, Ge Liu, Roger Ren. "SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning." arXiv preprint arXiv:2608.26432, 2026. https://arxiv.org/abs/2608.26432
Voice-agent benchmarks score a proprietary realtime API from the outside: no gradient crosses the boundary, and per-call pricing rules out the rollout volume RL needs. SpeechGym makes both sides of the conversation local open models talking in native speech over an existing tool-use benchmark, so a voice agent can actually be trained. A per-turn process reward then fixes the thing that breaks first: at audio success rates, almost every rollout group fails identically and teaches nothing.