A voice agent has to do everything a text agent does — call tools against a live database, respect a policy, drive a conversation to a verifiable end state — with speech as its only channel. The standard recipe trains the policy in text and bolts speech on at the edges. This note is about what happens when you stop doing that, and train the agent in audio end to end.
The loop, closed and local. The caller is not a recording and not a text simulator wired to a speech engine — it is an open model generating its turns as audio, inside the same process as the agent. That single choice is what turns an evaluation harness into a gym: gradients reach the policy, a rollout costs compute instead of API credit, and the reward is whatever the underlying benchmark already checks. Tasks, tools, database and success check are inherited unmodified, so the channel is the only variable.
The short version
The agent picks the right tool and the right argument slot, then fills it with a value it mis-heard from the waveform. That happens in 32% of speech rollouts against 2% in text — a sixteen-fold gap, and the single largest difference between the channels.
A perceptual error labels itself: a call with a mis-heard argument fails against the database while a correct one succeeds. The environment supplies that signal for free — no annotation, no reward model.
Near the floor almost every group of rollouts fails identically, so a group-relative objective sees no variance and produces no gradient: only 16% of groups carried one. Crediting each successful tool call raises that to 99.6%.
Run untuned inside an independently implemented benchmark — different user simulator, different TTS and ASR, different grader — the same checkpoint more than doubles pass@1, 24% → 53%, and an open 30B model moves from last place to second.
01 — What breaks
Run the same tasks, the same tools and the same reward down two channels, changing nothing but whether the caller types or speaks. Every failure mode gets worse in speech, but not evenly — and the one that moves most is not a reasoning failure at all.
Same tasks, same tools, same reward — only the channel differs. Mis-hearing is near-absent in text and ordinary in speech; the errors downstream of it are merely more common. That ordering is the whole diagnosis: the agent understands the task and picks the right call, then writes a digit of a zip code, a character of an order ID or a spelling of a name that it never heard correctly.
All three are one cascade, not three separate problems. A mis-heard value produces a wrong argument; the wrong argument produces a tool error; the error produces a retry of the identical call; and the retries burn the step budget until the episode times out at zero reward. An outcome-only view sees one undifferentiated failure. The environment sees four distinct events, each with its own signal.
A fourth failure, absent from the chart, has no counterpart in the text baseline: under an insistent caller the agent performs a state-changing write it was not authorised to perform, and ends the episode believing it helped. Pressure and urgency live in a voice and are flattened by a transcript. That one is reported qualitatively — there is no matched text number for it.
02 — The obstacle
Given a trainable loop and a labelled failure, the fix looks like ordinary RL. It is not, and the reason is arithmetic rather than audio. A group-relative objective normalises returns within a group of rollouts on the same task. When almost every episode fails, every rollout in the group returns zero, the standard deviation is zero, and the group contributes nothing — its compute is spent generating trajectories that never touch the weights.
Each square is one group of rollouts. Under outcome-only reward, 84 of every 100 groups are generated and then discarded — more than four fifths of the rollout budget producing no gradient at all. Crediting each successful tool call as it happens lets two rollouts that both failed still differ in how far they got, and almost every group becomes informative.
The shaping is deliberately unremarkable: +0.1 for each successful tool execution, −0.1 for each failed one, with the episode outcome appended to the final step so the terminal criterion is untouched. Four failing rollouts that scored {0, 0, 0, 0} under outcome-only reward might now score {0.30, −0.10, 0.20, 0.10} — still all failures, but now ordered, and an ordering is a gradient.
Note what this does not change. The reward that decides whether an episode succeeded is still the benchmark’s own check, run unmodified. The shaping only decides which rollouts are worth learning from along the way.
03 — It transfers
The obvious worry about any RL result is that the policy learned the environment rather than the task. So the trained checkpoint was run, with no further tuning, inside a separate evaluation-only voice benchmark built by a different group: its own cascaded user simulator, its own TTS voices over a telephony-grade channel, its own grader. Acoustics, user policy and scoring all change at once. Only the agent’s weights are ours.
Every pair comes from the same external pipeline. Same tasks, same user simulator, same acoustics, same grader, single attempt — only the weights differ. The gain holds in all three domains and the 95% confidence intervals do not overlap in any of them. The largest relative move is Telecom, the dual-control domain where the user also holds tools: a six-fold rise from a base of 4%.
Where that leaves an open model. The same 30B open-weights model moves from last place to second on this benchmark. The other systems are commercial stacks we did not build, train or re-run — their scores are as reported by their own providers, so read this as context rather than a controlled comparison. The controlled part is the highlighted pair, bottom and second from the top: identical model, identical benchmark, RL in between.
04 — Did it game the metric?
A jump from 24% to 53% invites the suspicion that a number was gamed rather than a task solved. Two checks, either of which could have falsified it.
Had the gain come from somewhere other than the cascade of §01, the rates that cascade produces would be roughly unchanged. They are not: every behaviour the environment was built to expose moved in the intended direction, measured on the raw trajectories of both models over the same external task set.
Inflating a success rate by spending more of the episode — retrying, or stalling until the user concedes — predicts turns and tokens rising alongside pass@1. They fall.
On the tasks the trained model actually fixes it is 25% more concise than the base was. Success rising while compute falls is the opposite of length-based reward hacking.
The annotations also record repair strategies nobody designed: asking the caller to spell a name out, retrying a lookup with a corrected spelling, switching the lookup key when the first one fails. Nothing in the reward mentions spelling, retries or lookup keys — it scores task completion and tool-call success. RL found them, consistent with raising the chance of finishing a task over an unreliable channel.
05 — What it costs to run
An episode here is dozens of turns, each generating audio from two 30B models. Rollout collection, not the gradient step, is the bottleneck — so serving those rollouts efficiently is what makes online speech RL practical at all.
Seven epochs. A run that costs over $1,400 in API calls costs nothing here, and the money was never the real obstacle — the rollout volume on-policy RL needs simply cannot be bought at that price.
Why the loop has to be local. On-policy RL needs to sample from the current policy, over and over. Behind a commercial voice API the gradients cannot cross the boundary at all, and even ignoring that, the per-rollout price rules out the volume: the paper puts one epoch at over $200, a seven-epoch run over $1,400, and a thousand-epoch budget near $200k. With both models open and local the same loop costs compute and nothing else.
Where the time goes. Batching the rollouts through an inference server cuts a full epoch from 15 hours to 2.8. The policy update is untouched at ten minutes — it was never the bottleneck. Multi-turn speech rollouts were, and they are the part that a commercial API also makes expensive.
06 — Scope
The user simulator is a model, not a person: it fixes the speech distribution that policy improvement is measured against — and an idealised one at that. Self-play audio is clean; the external benchmark supplies the noisier test, which is why the headline result is the one measured outside the training environment. Confidently-wrong over-action is reported qualitatively against text because there is no matched text baseline for it. Results are one model family on both sides of the conversation. And the commercial systems in the standing were evaluated by their own providers on their own deployed stacks — we did not build, train or re-run them.
Why it matters
The gap between what an agent can do in text and what it can do in speech has been reported as a property of the modality — a cost you pay for using voice. Closing the loop recasts it as something with an objective, a gradient and a stopping criterion: a deficit in identifiable competences — hearing a slot value correctly, confirming before acting, abandoning a failing plan — that closed-loop training substantially reduces. And because the interface is an ordinary reset/step/reward loop over text agentic domains, any open omni-modal model drops into it.
An audio language model takes a spoken question and answers it in one word. Between the sound going in and the word coming out sit forty-eight transformer layers we normally treat as a black box — and unlike a reasoning model, it writes nothing down on the way. This note is about what we found when we read those layers directly, while the model was still listening.
This is the model thinking, laid out whole. The entire input runs along the x-axis — the audio first, then the written question and its options — and all 49 readout depths run up the y-axis, input at the bottom. Pale blue is where the model is reading out a real word rather than punctuation or a control token, deeper inside the workspace band: its train of thought. Orange is wherever the pinned word appears. Hover any cell to see the words the model was closest to saying there. Pin each step of the chain in turn and watch the thought move — the early concept sits over the audio, the next one deepens, and the answer only commits at the far right, over the printed option.
The short version
Each one is stated with the number that carries it, and shown further down. If you read nothing else, read these.
An archaeology clip reconstructs a whole unspoken scene of a dig — burial, coffin, ruins, excavation, treasures — none of them in the audio, the question or the options. On some clips the model’s own transcription and its own caption both miss the decisive concept and the readout assembles it regardless. It is not re-transcribing.
Clip and prompt are entirely in English, yet one concept surfaces in several scripts at once — music / música / 音乐 / Musik / musica. Across 863,770 workspace readouts, 52.6% are English and 38.5% are Chinese. It holds a concept, not a word.
Given the same clip as audio and as the model’s own emotion-free caption, only the audio mind forms the true sound source, the speaker’s role, or the affect in a voice. Correct speaker role on 88.9% of disagreement clips versus 70.4% for the caption mind.
The audio-driven gap over the text prior is null at the input, turns on at about 12% depth, and is largest in the workspace band. Transplant real-audio activations into a run whose audio was deleted and the answer comes back — 10/10 patching early, 9/10 mid-stack, 0/10 if you wait for the last fifth.
Deleting the entry layer costs 65 points of accuracy. Deleting any single interior layer costs at most 10, and the band means are within a point of zero — the network routes around whichever one you remove.
01 — The hidden chain
The method is deliberately boring. Take the model’s own output head — the matrix it uses to turn a final hidden state into a word — and apply it early, at every layer, at the positions where the audio lives. That is the logit lens: no probe is trained, nothing is fine-tuned, and it costs one forward pass. What comes back is which word the model is poised to produce at that depth, long before any answer token exists. Everything below reads the workspace band — layers 17–38, the middle third to four fifths of the stack, taken a priori from the text-model literature and not tuned on this data.
What comes back is not a transcript. The readout carries concepts that appear in neither the question, the options, nor the model’s own written description of the clip. On an archaeology recording the workspace band reconstructs an entire unspoken scene of a dig before resolving to the archaeologist the question asks about. On a clip about a telephone patent it assembles telecommunications and wired on the way to the year.
The strongest cases go further: on some clips the model’s own two routes for turning audio into text — its verbatim transcription and its free-form caption — both miss the decisive concept, and the middle layers assemble it anyway. Two independent audio-to-text routes missing what the readout finds makes plain re-transcription an unlikely explanation.
The pattern is not unique to one clip. Pick any of these seven and read the chain it assembles, step by step, in the order each word becomes readable.
Each step is a word the readout reaches at the audio positions, ordered by the depth at which it becomes readable. Concepts in the middle of a chain appear in neither the question nor any option — they are inferred from the sound and then used. Where the chain ends in a name, that name commits at the output slot rather than over the audio; we mark that rather than overclaim it.
A lens can light up on generically common tokens, so the same readout was run for ten unrelated placebo words — banana, guitar, planet, umbrella, tuesday and others. Across ten clips the true answer concept fills 1–34% of workspace cells; the placebos fill 0.12%. Essentially never. The single visible exception is trumpet on the whip clip at 1.3%, where a whip-crack genuinely does resemble a brass transient.
The signal is also sparse in time. Unlike text, where readable content spreads over many tokens, only 4–30% of audio positions carry the concept — 5 of 34 positions on one clip, 10 of 37 on another. A randomly chosen audio position often reads as noise. That is why the readout aggregates by taking the best rank over positions rather than averaging, which would wash the signal out.
02 — A concept, not a word
Is the model quietly re-transcribing the sound, or holding an idea? On a music clip the notion of music appears in the workspace band as English music, Chinese yīnyuè, Spanish música, German Musik and Italian musica — all at once, at the same audio positions. On birdsong the leading readout is the Chinese niǎolèi. The model is not reaching for an English string it memorised; it is holding an idea and rendering it into whichever token happens to sit nearest.
Tallying the top-1 readout across all audio-region workspace cells — 863,770 cells over 948 clips — the inner vocabulary is 52.6% English, 38.5% Chinese and at most 4.3% anything else. More than a third of the model’s leading guesses are in a language that appears nowhere in its input.
If the model had memorised an English string, the readout would be English. Instead one concept surfaces in several scripts at the same audio positions, most of them at rank 1. It is holding an idea and rendering it into whichever token sits nearest.
The inputs are entirely in English. This is the language of the top-1 readout at the audio positions across the whole corpus — 948 clips, 863,770 workspace cells, lens-noise fragments dropped. More than a third of what the model is poised to say is Chinese. The paper reports the two leading figures, 52.6% English and 38.5% Chinese, and bounds everything else at 4.3%; the per-language split below that ceiling comes from the project’s own tally of the same cells.
That is not the lens’s known fondness for frequent CJK tokens. A frequency control shows the Chinese form is clip-specific: yīnyuè is rank 1 on 30% of music clips but only 6% of the 610 non-music clips — a 5× lift — and Chinese bird forms show a 9–20× lift on birdsong, where a globally frequent token would lift near 1.
03 — Spreading activation
Beyond the task-relevant concept, semantic neighbours of what the model hears light up spontaneously — spoken nowhere in the clip, in no question and no option — and are then set aside. On a skateboard clip a single audio position reads down the layers as a schedule: the raw sound is verbalized (gǔndòng, “rolling”), the category forms (skate / skateboard), and only then a neighbour ignites — huáxuě, skiing, a sibling board-sport — held for several layers, never rank 1, then dropped. The model answers Skateboard.
Across every family we could verify, the associate always follows its evoker, and the farthest one (老虎, tiger) lands latest at about 92% depth. The richest cases are whole neighbourhoods: an archaeology clip reconstructs a dig scene — burial, coffin, ruins, excavation, treasures — none of them spoken in the clip. And this is largely not a geometry artifact of the output embedding: five of six associates fall outside their evoker’s top-100 cosine neighbours, and the one near pair (skiing, the 7th neighbour of skate) still ignites strictly after its evoker and is then dropped — a schedule that static similarity does not encode. It is the audio analogue of spreading activation, read layer by layer, with depth playing the role of time.
Every one of these is spoken nowhere in the clip — and in neither the question nor any option. Box size scales with how often the word fires in this clip; the second figure is how many of the 948 corpus grids contain it at all. All of them are rare, so this is genuine association rather than a globally common token.
Four phases: verbalize, categorize, ignite, dismiss. The rolling sound is put into words first. The category arrives at L32. Only then does 滑雪 — skiing, a sibling board-sport named nowhere in the clip — ignite beside it, ride along for several layers without ever reaching rank 1, and drop out of the readout entirely by L46. The model answers Skateboard.
The associate never comes first. Across every family we could verify, the neighbour becomes readable at or after its evoker — and the farthest, evidence-free one (老虎, tiger) lands latest at about 92% depth. Two clocks are visible: a perceptual competitor rides the evidence and arrives early but only on the frames that support it — wolf is readable from ~38% and fires on the howl frames, not the growl frames — while a memory associate needs no acoustic support and arrives late. Both are set aside; the model answers Lion.
04 — What words throw away
Sound carries how something was said — an emotion, a speaker’s role, an acoustic source — that no transcript preserves. So feed the model the same clip twice: once as the waveform, once as the model’s own emotion-free caption of it, with the question and options identical in both runs.
On the clips where the two minds disagree, only the audio mind forms the true, speech-specific concept. A roar reads lion and the model answers Lion; the caption run never forms the concept and answers Wolf. A clip reads priest where the caption hears only “father”. A sarcastic voice reads the inferred affect frustration, a word named nowhere in the prompt and nowhere in the caption.
Two minds, one clip, side by side. Every column is a token position across the whole prompt — the system preamble, then the audio, then the question and the printed options — and every row a sampled depth. The pale layer shows where the readout is a real word at all; the marked cells are where it is the concept lion. Given the waveform the concept lights up across the audio span and through the workspace band, and the model answers Lion. Given only its own written description of the same clip — which says “deep growls” and “howl” but never names the animal — the concept barely forms at all, and the model answers Wolf. What a sound is lives in the model’s mind only while it can hear it.
Hearing does not always help. On 500 spoken TriviaQA questions the answer is a stored fact and the spoken clip carries words identical to the written question — the sound adds nothing the text does not already supply. There the text mind wins: reading both minds on the 87 items it gets right from text and wrong from speech, the answer concept appears in the text mind’s workspace on 83% of them versus 62% for the speech mind. We treat that task as the boundary of the effect, not as a clean mechanism.
05 — Where the thought lives
This is the control the whole paper rests on. The question and the four printed options are byte-identical in all three runs — only the sound is swapped. If the readout were just a prior over the printed options it would not move. It moves all the way to chance. Accuracies are the workspace band of Table 2; ranks are the threshold-free median over the 152,064-token vocabulary.
The whole map on one axis. Nothing the sound adds is answer-relevant through the first eighth of the network. The gap over the text prior then turns significant and never closes; the answer concept stabilizes about a third of the way up and holds; and by the motor layers the decision is already made — patching there recovers nothing. Depths are measured, not assumed.
The findings above read the workspace band by assumption — layers 17–38, taken a priori from the text-model literature, not tuned on this data. So the next question is whether the audio-driven signal actually lives there. Sweeping the readout band by band, the real−silence gap — the part of the answer the printed text alone cannot supply — is negligible in the sensory band and opens up sharply in the workspace.
Threshold-free, the same ordering shows up in the raw rank of the correct answer among all 152,064 vocabulary tokens: #2,297 with the real clip, #5,040 with a mismatched one, #26,241 with silence. And reading that gap layer by layer, the two conditions are statistically indistinguishable through the first several layers — the encoder has deposited acoustic features, but nothing the sound adds is answer-relevant yet — and then, a little past a tenth of the way up, the gap opens at about 12% depth and stays open. Thirty-seven of 49 readout depths reach significance, 35 surviving multiple-comparison control, concentrated in the workspace band (18 of 22 layers, versus 9 of 17 sensory).
Decodability need not mean the model uses what we can decode. So: take clips the model gets right with audio and wrong once the audio’s features are zeroed, then copy the clean run’s activations into the corrupted run at one band and let it finish.
So the audio content is causally used, and committed before the last fifth of the network. And it is not a quirk of one checkpoint: repeating the band-localized audio-swap on an architecturally different model — Qwen2.5-Omni-7B, a dense 28-layer Thinker against our sparse mixture-of-experts — reproduces the same signature, with the gap small in the sensory band and large in the workspace (p < 10−3).
06 — The functional map
Reading is correlational; deleting is causal. Bypassing one layer at a time — replacing its output with an identity skip — and measuring accuracy over 40 clips gives a clean functional map.
Only the entry layer is irreplaceable for accuracy. Deleting the layer where the audio is read in costs 65 points of accuracy — so far off the scale that it is drawn separately above. Every one of the other 47 layers costs at most 10 points, several cost nothing, and 20 of them slightly help. Averaged by band, and leaving that entry layer out, no band moves as much as a point and a half: sensory −0.2, workspace +1.1, motor 0.0. Retrieval is not carried by any one layer — the network routes around whichever one you remove.
Breaking accuracy is one thing; which ability breaks is another. Tracking three separable functions per deleted layer on one clip: perception fails only for layers 0–1, delivery fails only for layer 47, and no single layer stops the answer concept from surfacing at all. The pipeline is therefore: read the sound in (L0–L1), hold and recall it across the interior, deliver it out (L47) — and only the two ends are irreplaceable.
The same readout localizes errors. Take the clips the model answers correctly from real audio but wrongly from its own caption — the caption dropped the decisive acoustic detail, so the text-only mind falls back on a plausible neighbour. Across 73 such clips, the depth at which the wrong concept first takes over is early: median onset 12% depth, 58.9% formed by 20% depth, 46 of 73 inside the sensory band, and 84.9% commit before the output layer rather than slipping at the end.
A text-mode hallucination is thus typically not a late slip. The model locks onto a plausible-but-wrong concept at roughly the same depth where correct answers first emerge, and then never revises it. Reading the two minds side by side both detects the hallucination — the audio mind holds the right concept where the text mind holds the wrong one — and shows where it entered.
07 — Scope
The account is deliberately conservative, and the limits are worth stating plainly rather than burying. The logit lens is a cheap proxy, not a faithful one — against a faithful lens our readouts should, if anything, under-detect, and early-layer readouts are noisy by construction, which is why the claims concern the middle band. This is one model family, on a sound-dominated slice of clips, with a signal that is sparse across audio positions. The patching test localizes coarsely: it shows the content is committed before the motor band, not that the workspace band alone is responsible.
And “workspace” is used in the functional, information-posted-for-report sense from cognitive science. No claim is made about subjective experience. The quantities reported here are controls — they exist to rule out the model reading a prior off the printed options — not benchmark scores.
Why it matters
Monitoring a model’s stated reasoning only works if the model states it — and a speech agent answering a spoken question directly writes nothing down. Worse, stated reasoning need not be faithful. What this readout shows is that in an audio LLM the decision-relevant concepts are already legible, in words, before a single token is emitted: multilingual, paralinguistic, free-associating, and causally used.
Two uses follow if that holds under a faithful lens. As a training signal, a workspace readout offers a process-level target where a scalar reward sees only the final answer. As a monitor, it is close to free — one matrix product per position read — against deployed multimodal systems that are already latency- and memory-bound. The natural next test is whether safety-relevant decisions — tool calls, refusals, fabrication — are readable this way before a speech agent acts.
Chain-of-thought is the reflex answer for making a model smarter. In Audio LLMs it backfires: tell the model to think first and it gets worse, and the longer it thinks the worse it gets. This note is about why that happens, why it is not an argument against reasoning, and what changes when you supervise the reasoning process instead of only the final answer.
Flip the switch. Every number is MMAU Test-mini total accuracy from Table 1 of the paper — the left state is the model answering directly, the right state is the same model told to reason first. The untrained base model falls 3.40 points for thinking. After training on the reasoning process, CESAR gains 3.40 for it: the same margin, sign reversed. That reversal is what “solving test-time inverse scaling” means.
The short version
Each is stated with the number that carries it, and shown in full further down.
Ask Qwen2.5-Omni-7B to think before answering and MMAU accuracy falls from 68.60 to 65.20. Longer chains kept making it worse. We name this test-time inverse scaling.
Models never taught how to reason produce hallucinatory, inconsistent chains whose errors compound. The capacity is there; the supervision was not.
Outcome-only RLVR scores the final token and ignores the road to it. CESAR’s reward has five terms, three of them process-level — reasoning–answer consistency; structure, logic and domain grounding in one term; and a penalty on overthinking.
Under the same test, CESAR gains +3.40 from reasoning — exactly the margin the base model lost. Reasoning stops being a liability and becomes the source of the gain.
77.10 on MMAU Test-mini against 71.60 and 62.50, and the top-scoring 7B model on the harder MMAU-Pro at 56.4.
On MMSU, reasoning reaches 81.07 against a human 86.77 — and perception rose too, though it stays far behind humans. That gap is the real bottleneck.
01 — The paradox
Text LLMs made chain-of-thought look like a free lunch: o1 and DeepSeek-R1 turned longer deliberation into better answers. Carrying the same prompt into audio produces the opposite. On the open Audio LLM this work starts from, switching reasoning on costs 3.40 points, and an outcome-only RL baseline recovers only to break-even; sweeping the maximum thinking length makes the loss deepen rather than recover. We call it test-time inverse scaling, and it is the first thing a process-level view has to explain.
02 — The diagnosis
It would be easy to read that as evidence that audio reasoning is a dead end. The chains themselves say otherwise. Supervised fine-tuning on CoT data teaches a model to imitate the shape of reasoning; outcome-only RL rewards it only for landing on the right option. Neither ever looks at the middle. Three failure modes follow directly, and all three are visible in the traces.
03 — The fix
CESAR keeps GRPO and keeps verifiable correctness, then stops treating the reasoning trace as a black box between prompt and answer. Five terms make up the total reward. The first two are the conventional verifiable pair; the last three are the ones that make reasoning a trainable skill rather than an emergent accident.
Training is GRPO on Qwen2.5-Omni-7B, 8 sampled responses per example, on AVQA augmented with answer-invariant rephrasings so the model has to learn the reasoning rather than the wording.
The whole of Table 1, not just the headline. Switch tracks and the pattern holds: on every one of the four, the base model loses ground by reasoning and CESAR gains it. Sound is where the gap is widest — 83.48 against the base model’s 69.07 — and Speech is where the overthinking penalty costs something, which is why the variant without it leads there. Proprietary systems are shown for scale and have no reasoning switch to flip.
04 — What changed
The headline is not that accuracy went up. It is that the relationship between thinking and accuracy inverted — the mirrored bars at the top of this page. What follows from that inversion is the part that matters for anyone choosing a model.
05 — The sweet spot
Sweeping the maximum thinking length from 0 to 250 tokens separates the two regimes cleanly. Baselines either collapse or wander with no reliable gain. Our variant without the overthinking penalty climbs steadily to a 76.50% peak. The full method, penalised for rambling, peaks higher at 77.10% using a chain of only about 35–40 tokens. The penalty is not a tax on thinking; it is what teaches the model when to stop.
06 — The ceiling
MMSU separates what a model hears from what it concludes, and the split is stark. CESAR’s reasoning lands within six points of expert humans — and beats them outright on semantic reasoning, 88.72 against 82.16. Perception is another story: 48.45 against a human 91.24. Training the process lifted perception too, which is the genuinely surprising part, but the gap that remains is the one that will decide how far audio reasoning can go.
Reasoning is close to human; perception is nowhere near. Each point is one system on MMSU, reasoning on the horizontal axis and perception on the vertical. Every model sits far below the dashed parity line — they all reason far better than they hear, and only the human reference clears it. CESAR posts the best reasoning score of any model here, 81.07, within six points of the human 86.77 — while its perception, at 48.45, is 42.79 points short of the human 91.24. That asymmetry is the finding: process rewards move reasoning close to human parity and lift perception with it, but the perceptual bottleneck is what remains.
07 — The verdict
Accuracy says the answers improved. It cannot say the reasoning did. So the full 1,000-question MMAU Test-mini set was judged by three independent expert annotators — over 3,000 individual judgements, blind to which model wrote which trace and blind to the correct answer, scoring only which reasoning process was sounder.
08 — What it means
The tidy reading of test-time inverse scaling was that audio is simply not a reasoning-friendly modality. The result here says something narrower and more useful: what fails is reasoning nobody trained. Give the process its own reward — consistency with its own conclusion, analytical structure, domain grounding, and a real cost for rambling — and the same 7B model that lost 3.40 points to thinking gains 3.40, passes systems many times its size, and reasons within touching distance of expert humans.
The wall it runs into next is not cognitive. It is perceptual: 48.45 against 91.24. The next gain in audio reasoning will not come from thinking harder about what the model heard. It will come from hearing it better.
RL fine-tuning of a generative model runs on one dial. Turn it up and the model keeps its diversity but stops improving; turn it down and it chases reward until it collapses into template output. The dial is a single coefficient applied identically to every sample. This note is about what happens when you stop treating it as a constant.
βtot = β0 − A+0.00 = 1.00
Average sample. The advantage is near zero, so the coefficient stays at its baseline 1.00 and the sample barely moves the policy.
One subtraction, and the dilemma dissolves. Every RL fine-tuning method before this one picks a single β and applies it to every sample: raise it and the model is protected but cannot improve, lower it and it improves until it collapses. ADRPO — Adaptive Divergence Regularized Policy Optimization — makes β a function of the sample — Equation 4 in the paper is literally βtot = β0 − A. Good samples get freedom, bad samples get held down, and nobody has to tune the trade-off. β0 = 1 with A clipped to [−1, 1] is the paper’s text-to-image setting.
The short version
Each is stated with the number that carries it, and shown in full further down.
Strong regularisation protects the pre-trained model but caps the reward. Weak regularisation chases reward and invites collapse or hacking. Every sample gets the same β, whether it deserves it or not.
ADRPO sets β = β0 − A. High-advantage samples are regularised less and exploited harder; low-advantage samples are pulled back toward the reference model. One term, no extra networks, no architecture change.
Reward-weighting can only give a bad sample a small positive push. An advantage-weighted objective flips the gradient sign and pushes the policy away from that sample — actively suppressing poor generations instead of politely ignoring them.
Fine-tuned SD3 at 2B parameters outscores FLUX.1-Dev (12B) and SANA-1.5 (4.8B) on ClipScore, aesthetics and human preference. Adaptive regularisation bought more than a 6× parameter increase would have.
Every competing method trades one for the other. ADRPO finishes at 5.13 diversity against the base model’s 5.08 — level to within seed noise, while posting the highest alignment score in the table.
On Qwen3, ADRPO escapes a local optimum by deliberately raising entropy and converges at 5× GRPO’s reward. On MMAU a 7B model reaches 76.0, past Gemini 2.5 Pro and GPT-4o Audio.
01 — The dilemma
Every mainstream RL fine-tuning objective — PPO, GRPO, DPO, and W2-regularised flow matching — carries a divergence penalty scaled by a fixed β. That single number is asked to do two opposing jobs simultaneously: keep the policy near the pre-trained model so it does not forget or collapse, and get out of the way so the policy can actually improve.
The bind is that the right answer differs per sample. A generation the reward model scores highly is a direction worth committing to; holding it back is pure loss. A poor generation is exactly where you want the reference model’s pull to be strongest. A constant cannot express that, so practitioners tune β to a compromise that is wrong for both cases.
02 — The fix
ADRPO replaces the constant with β = β0 − A, where A is the advantage estimate already being computed for the policy gradient. Nothing is added to the model, no second network is trained, and the objective stays a drop-in for existing methods. What changes is that the constraint now varies inversely with sample quality.
03 — Text-to-image
ADRPO was applied to SD3 (2B parameters) on DrawBench prompts with CLIP score as the reward, against offline DPO, reward-ranked RAFT, fixed-β ORW-CFM-W2, and two much larger models that were never RL-tuned at all. It leads on task metrics, image quality and human preference at the same time.
Six metrics, one table, nothing cherry-picked. The red line is the un-tuned SD3 the fine-tuned rows all start from. Click through the metrics and the shape of the problem shows: RAFT falls behind the base model on four of the six, ORW-CFM-W2 buys alignment and pays in diversity, and ADRPO is the only row that is at or above the base model on every one — and it leads all six outright. Two of the rows are larger models — 6× and 2.4× its parameter count.
04 — The Pareto front
This is the result that is hard to get by tuning a constant. Reward-ranked fine-tuning buys almost no alignment at all (29.35 against the base model’s 29.27) and still flattens the output distribution to pay for it — diversity falls from 5.08 to 1.85, a textbook collapse. Fixed-β W2 regularisation is gentler but still pays 3.86. ADRPO finishes at 5.13 against the base model’s 5.08 — unchanged to within the reported seed noise (±0.47 and ±0.52) — while posting the highest alignment score in the table. It is the only row that buys alignment without paying for it.
05 — Language models
The same objective drops into GRPO for LLM fine-tuning by making the KL coefficient advantage-dependent. Tracked in reward–entropy space on Qwen2 (0.5B) and Qwen3 (0.6B), with RM-Gemma-2B supplying the reward, ADRPO and GRPO take visibly different paths. GRPO holds high entropy throughout and moves sideways — lots of exploration, little reward found. ADRPO first tightens into a low-entropy region, then deliberately raises entropy again to break out of the local optimum it landed in, and converges at 5× GRPO’s final reward.
Nobody designed that behaviour. It falls out of the coefficient: once a region stops producing advantage, regularisation rises, the policy loosens, and exploration resumes on its own. The absence of that mechanism is why GRPO’s later checkpoints often score worse than earlier ones while ADRPO improves monotonically — no early stopping required.
06 — Audio reasoning
Continuous flow matching and discrete token generation are different enough that a shared mechanism is worth testing on a third case. Qwen2.5-Omni-7B was fine-tuned on AVQA with verifiable rewards and evaluated on MMAU.
07 — Robustness
Adaptive regularisation introduces a clipping range for the advantage. If performance were delicately balanced on it, the method would have traded one tuning problem for another. Sweeping it over a 4× span moves the total by less than half a point, and every setting still beats fixed-β GRPO.
08 — What it means
Exploration versus exploitation is usually framed as something you resolve before training by picking a coefficient, and live with afterwards. The result here is that the information needed to resolve it properly is already sitting in the training loop: the advantage estimate says, for this particular sample, whether the policy has found something worth committing to. Subtracting it turns a global compromise into a local decision.
What makes the result more than a tuning trick is the range it survives. The same one-term change holds across continuous flow matching with a Wasserstein penalty, discrete LLM generation with a KL penalty, and multi-modal audio reasoning — three architectures, two divergence measures, one subtraction, and a 2B model that outperforms a 12B one.