Chain-of-thought prompting has transformed text-based LLMs. Models like OpenAI o1 and DeepSeek-R1 show that explicit reasoning dramatically improves performance. But something counterintuitive happens in Audio LLMs: more reasoning makes things worse. This post explains why, what’s actually going wrong inside the reasoning chains, and how process-level rewards resolve the issue.

Table of Contents

• The Scaling Paradox
• Why Reasoning Goes Wrong
  • The training gap
  • Three failure modes
  • Why doesn’t reasoning emerge naturally?
• Process Rewards: Supervising How the Model Thinks
  • Reasoning consistency reward
  • Structured reasoning keywords
  • Overthinking penalty
• The Reasoning Sweet Spot
• Beyond Reasoning: Synergistic Effects
• Broader Lessons
  • 1. Process supervision matters
  • 2. More compute is not automatically better
  • 3. Simple rewards can be powerful
• Summary
• References

The Scaling Paradox

In text LLMs, there’s a reliable pattern: allocating more compute to reasoning at inference time (longer chain-of-thought, more tokens to “think”) tends to improve performance. This is the foundation of test-time scaling — the principle behind reasoning models like o1, R1, and QwQ.

Audio LLMs tell a different story. When you allow an Audio LLM to reason before answering, performance often degrades. When you let it reason longer, it gets even worse:

where \(P(L_{\text{max}})\) is accuracy when the model is allowed up to \(L_{\text{max}}\) reasoning tokens.

We call this test-time inverse scaling: the more the model “thinks,” the worse it performs.

This isn’t subtle. Models that achieve 70%+ accuracy with direct answers can drop to below 60% when forced to reason. The reasoning itself is actively harmful.

The natural conclusion might be: “reasoning just doesn’t work for audio.” But that’s wrong. The problem isn’t reasoning itself — it’s how models are trained to reason.

Why Reasoning Goes Wrong

The training gap

Current RL approaches for Audio LLMs use outcome-only rewards:

The reward checks two things: “Is the final answer correct?” and “Is the output properly formatted?” That’s all. The entire reasoning chain \(t_i\) — which could be hundreds of tokens — is completely invisible to the reward function.

This means the model gets the same reward whether its reasoning is:

"The audio contains a clear piano melody in the major key, with a tempo around 
120 BPM. The harmonic structure suggests classical rather than jazz..."  → Answer: B

or:

"Let me think about this. I think the answer might be something. Actually let me 
reconsider. The sound is interesting. Maybe it could be several things..."  → Answer: B

As long as the final answer is correct, both receive identical reward. The model has zero gradient signal to improve reasoning quality.

Three failure modes

Without supervision of the reasoning process, three specific pathologies emerge:

1. Hallucinatory reasoning. The model generates plausible-sounding but factually wrong analysis — describing instruments that aren’t present, inventing acoustic properties, confabulating music theory. Each additional reasoning token is another opportunity to introduce a hallucination, and hallucinations compound across the chain.

2. Reasoning-answer inconsistency. The reasoning says one thing, but the answer says another. Example:

“The audio clearly contains a piano solo… therefore the answer is drums.”

With outcome-only rewards, the model can learn to produce correct answers despite wrong reasoning. The reasoning becomes decorative noise disconnected from the answer. Longer chains amplify this noise.

3. Unstructured, circular reasoning. Without guidance on how to reason, models develop rambling patterns: repeating observations, circular logic, tangential elaboration. No systematic analysis structure (elimination, comparison, deduction) emerges.

Why doesn’t reasoning emerge naturally?

In text LLMs, reasoning does emerge somewhat naturally from outcome-only training (though process rewards still help). Why is audio different?

The key difference is the modality gap. Text reasoning can leverage massive amounts of pre-training data that already contains reasoning patterns (textbooks, proofs, tutorials). The model has seen millions of examples of structured argumentation.

Audio reasoning has no such foundation. Pre-training on audio-text pairs teaches the model to describe audio, not to reason about it. When you then ask it to reason via RL, it has no templates to draw from — so it generates text that looks like reasoning but isn’t grounded in actual audio analysis.

Process Rewards: Supervising How the Model Thinks

The fix is to reward not just the final answer, but the reasoning process itself. CESAR introduces a multi-faceted reward suite:

Each process reward targets a specific failure mode:

Reasoning consistency reward

This measures semantic alignment in two directions:

• Reasoning ↔ Answer (\(\text{Sim}(t_i, \hat{y}_i)\)): Does the reasoning logically lead to the chosen answer? This directly combats reasoning-answer inconsistency.

• Reasoning ↔ Question (\(\text{Sim}(t_i, Q_i)\)): Is the reasoning actually about the question asked? This prevents hallucination — the model can’t ramble about unrelated topics.

Similarity is computed via concept overlap:

This is deliberately simple — no learned model, no LLM judge, just lexical overlap of key concepts. The simplicity is a feature: it provides a stable, deterministic gradient signal without introducing the complexity and variance of learned reward models.

Structured reasoning keywords

Three components that serve as cognitive scaffolding:

Analytical patterns (\(R_{\text{pattern}}\)): Detects and rewards structured reasoning strategies — elimination (“Option A is unlikely because…”), comparison (“Comparing the tempo to…”), step-by-step analysis (“First, I’ll analyze the frequency spectrum…”). These patterns don’t emerge reliably from outcome-only training.

Logical connectives (\(R_{\text{logic}}\)): Rewards markers of genuine logical reasoning — causal chains (“therefore,” “because”), hypotheticals (“if the tempo were…”), evidence-based conclusions (“based on the harmonic content…”). Promotes coherent logical progression rather than stream-of-consciousness.

Domain vocabulary (\(R_{\text{domain}}\)): Rewards audio-specific terminology — acoustic properties (timbre, frequency, amplitude), musical concepts (tempo, key, chord progression), speech features (prosody, formants). This grounds reasoning in the actual audio signal rather than generic text patterns.

Overthinking penalty

A linear penalty on reasoning length. This is the essential counterpart to keyword rewards — without it, the model could game the keyword reward by producing verbose, repetitive reasoning stuffed with trigger words.

The penalty forces the model to be concise. Combined with the keyword reward, the optimization pressure becomes: “reason in a structured, domain-grounded, logically coherent way — and do it efficiently.”

The Reasoning Sweet Spot

With process rewards, test-time inverse scaling reverses. Performance now increases with reasoning length, up to a model-specific optimum:

CESAR discovers “reasoning sweet spots” where the model achieves peak performance. Below this point, the model lacks sufficient analytical depth. Beyond it, returns diminish (but don’t degrade — a critical difference from baselines).

The scaling slope — whether performance increases or decreases with more reasoning — is perhaps the most important metric. A positive slope means the model has genuinely learned to reason. A negative slope means it’s just generating noise that happens to sometimes contain correct answers.

Beyond Reasoning: Synergistic Effects

An unexpected finding: improving reasoning quality also improves perception capabilities. Models trained with process rewards become better at basic audio understanding tasks (identifying instruments, recognizing speakers, detecting events) — even though these tasks don’t require explicit reasoning.

This suggests that reasoning and perception are deeply entangled in multimodal models. Learning to reason about audio signals forces the model to develop better internal representations of those signals. The process rewards act as an implicit regularizer that shapes how the model attends to and processes audio features.

Broader Lessons

1. Process supervision matters

The parallel to education is direct. If you only grade students on final exam answers, some will learn the material and some will learn to guess well. If you also evaluate their work (proofs, derivations, reasoning steps), you select for genuine understanding. Outcome-only rewards are like grading only final answers.

2. More compute is not automatically better

Test-time inverse scaling is a cautionary tale for the “just scale it” mentality. Without proper training, additional inference-time compute is wasted or harmful. The bottleneck is not compute — it’s the quality of what the model does with that compute.

3. Simple rewards can be powerful

CESAR’s process rewards are deliberately simple — concept overlap, keyword detection, length penalty. No learned reward models, no LLM judges, no complex architectures. The power comes from providing any gradient signal on reasoning quality, where previously there was none. Going from zero supervision to basic supervision of the reasoning process is the critical step.

Summary

The key insight: test-time inverse scaling is a training problem, not a fundamental limitation of reasoning in audio. Process rewards resolve it by providing the gradient signal that outcome-only rewards lack.

References

[1] Fan et al. “Incentivizing Consistent, Effective and Scalable Reasoning Capability in Audio LLMs via Reasoning Process Rewards.” ICLR 2026. Paper

[2] Shao et al. “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.” 2024. (GRPO)

[3] Wei et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” NeurIPS 2022.

[4] OpenAI. “Learning to Reason with LLMs.” 2024. (o1)

[5] DeepSeek-AI. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.” 2025.

Cited as:

Fan, Jiajun. "Test-Time Inverse Scaling in Audio LLMs."
jiajunfan.com (2025).

RLHF fine-tuning of generative models faces a fundamental tension: you want to maximize reward (make outputs better), but doing so aggressively destroys the diversity of what the model can produce. This post explains why this happens, why it’s hard to fix with a single hyperparameter, and how advantage-based adaptive regularization provides a principled solution.

Table of Contents

• Background: The RLHF Objective
• The Diversity Collapse Problem
  • What happens without regularization
  • Why diversity matters
  • The standard fix: fixed regularization
• The Fixed-β Dilemma
• Adaptive Divergence Regularized Policy Optimization (ADRPO)
  • Core idea
  • For flow matching models
  • For language models
• What Emerges in Practice
  • Smaller models outperform larger ones
  • Emergent exploration in LLMs
  • Cross-domain generality
• Summary
• References

Background: The RLHF Objective

Suppose you have a pre-trained generative model \(\pi_{\text{ref}}\) — a text-to-image model like Stable Diffusion 3, or a language model like Qwen. You want to fine-tune it so that its outputs score higher on some reward function \(R(x, c)\), where \(x\) is the generated output and \(c\) is the conditioning context (e.g., a text prompt).

The standard approach formulates this as a regularized optimization problem:

The first term says “produce high-reward outputs.” The second term says “don’t stray too far from the original model.” The coefficient \(\beta\) controls this trade-off, and \(D\) is some divergence measure — KL divergence for language models, Wasserstein-2 (W2) distance for flow matching models.

This is clean and intuitive. But in practice, getting \(\beta\) right is surprisingly hard — and the consequences of getting it wrong are severe.

The Diversity Collapse Problem

What happens without regularization

Let’s start with the extreme case: \(\beta = 0\), no regularization at all. The model is free to do whatever maximizes reward.

In online RL, the model generates samples, gets rewards, and updates itself. Without any constraint:

1. Early in training, the model discovers a few high-reward outputs
2. It shifts probability mass toward those outputs
3. Now it generates similar outputs more often → they get reinforced further
4. This positive feedback loop concentrates all probability mass on a narrow region

The equilibrium is a delta distribution: the model always generates the same output for a given prompt, regardless of the diversity of valid responses.

In image generation, this means every prompt produces essentially the same “optimal” image — perhaps technically high-scoring but boring and unusable. In language, the model repeats the same phrasing over and over.

This is mode collapse (or diversity collapse), and it’s the central failure mode of unconstrained RLHF.

Why diversity matters

One might ask: if the goal is to maximize reward, who cares about diversity? Several reasons:

1. Reward functions are imperfect proxies. They approximate human preferences, not capture them fully. A collapsed model that exploits quirks in the reward function (reward hacking) produces outputs that score high but look terrible to humans.

2. Generalization. A diverse model handles novel prompts gracefully. A collapsed model performs well only on prompts similar to its training distribution.

3. Downstream utility. In creative applications (art, writing, brainstorming), diversity is itself valuable. A model that always gives the same answer is useless for creative tasks.

4. Training stability. Once the model concentrates on a narrow manifold, gradient signals become noisy and training destabilizes.

The standard fix: fixed regularization

The conventional solution is to set \(\beta > 0\). For flow matching models, we typically use a W2 regularization term:

For language models with GRPO:

This works — to a point. But it creates a fundamental dilemma.

The Fixed-β Dilemma

Here’s the core problem: different samples need different levels of regularization.

Consider two samples generated during training for the same prompt:

Fixed \(\beta\) applies the same regularization pressure to every sample regardless of quality. If \(\beta\) is high enough to prevent collapse on bad samples, it also unnecessarily constrains good samples. If \(\beta\) is low enough to exploit good samples, it fails to stabilize bad samples.

This is not a tuning problem — no single value of \(\beta\) is optimal for all samples simultaneously.

Adaptive Divergence Regularized Policy Optimization (ADRPO)

Core idea

The insight is simple: make regularization strength a function of sample quality. High-quality samples get less regularization (exploit); low-quality samples get more (explore safely).

The general ADRPO objective:

where \(A\) is the advantage estimate for the current sample, \(\beta_0\) is a baseline coefficient, and \(\mathcal{L}_D\) is the divergence regularization loss.

The effective regularization coefficient becomes:

This is a one-line modification to existing RLHF objectives. No new networks, no complex architecture changes.

For flow matching models

For flow matching models like SD3, ADRPO combines advantage-weighted flow matching with adaptive W2 regularization:

Notice that \(A\) appears in both terms. In the first term, the advantage provides bidirectional learning signals:

• \(A > 0\): gradient encourages matching the target velocity (strengthen good generation)
• \(A < 0\): gradient reverses, actively pushing the model away from bad generation

This is fundamentally different from reward-weighting (ORW-CFM-W2), which can only down-weight bad samples with non-negative weights but never actively suppress them. In high-dimensional spaces like image generation, where bad regions vastly outnumber good ones, this difference is critical.

The advantage is estimated simply as:

i.e., the reward minus the batch-average reward for the same prompt.

For language models

For LLMs, ADRPO integrates with GRPO by making the KL coefficient advantage-dependent:

where \(A_{\text{GRPO}}\) is the group-level advantage from GRPO (reward minus group mean, normalized).

What Emerges in Practice

Smaller models outperform larger ones

With ADRPO, a 2B parameter SD3 model outperforms FLUX.1-Dev (12B) and SANA-1.5 (4.8B) in attribute binding, semantic consistency, and compositional control. The adaptive regularization extracts more from each parameter by allocating exploration budget where it matters most.

Emergent exploration in LLMs

When applied to LLM fine-tuning, ADRPO exhibits an unexpected emergent behavior: the ability to escape local optima.

When the model is stuck in a poor solution (all advantages near zero or negative), the adaptive coefficient \(\beta_0 - A\) increases globally, pulling the model back toward the pre-trained distribution — effectively “resetting” exploration. When it finds a promising direction (high advantages), regularization drops, allowing rapid exploitation. This creates a natural curriculum that no fixed coefficient can replicate.

Cross-domain generality

ADRPO is not limited to one modality or one divergence measure. It has been validated across:

• Flow matching (SD3) with W2 regularization
• Text LLMs (Qwen2, Qwen3) with KL divergence
• Audio reasoning LLMs with GRPO

The same principle — advantage-based adaptive regularization — provides consistent improvement regardless of the underlying architecture.

Summary

The exploration-exploitation dilemma in RLHF is fundamental: no single regularization coefficient is optimal for all samples. ADRPO resolves this by making \(\beta\) a function of advantage:

One line of math; dramatic practical consequences. For details, see the paper (NeurIPS 2025).

References

[1] Fan et al. “Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models.” NeurIPS 2025. Paper

[2] Fan et al. “Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization.” ICLR 2025. Paper

[3] Shao et al. “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.” 2024. (GRPO)

[4] Esser et al. “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis.” ICML 2024. (Stable Diffusion 3)

[5] Rafailov et al. “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” NeurIPS 2023. (DPO)

Cited as:

Fan, Jiajun. "The Exploration-Exploitation Dilemma in RLHF for Generative Models."
jiajunfan.com (2025).