TL;DR — RL with verifiable rewards teaches a tool-using agent what a correct call looks like, but never how to sequence calls. We add a procedural reward — invocation-order quality plus a final-system-state check — and an error-aware penalty to GRPO. A 3B model then beats correctness-only RL on all three benchmarks, and lands in the same range as much larger proprietary models.
In state-aware environments the order of tool calls decides whether the final system state is right. Wrapping a gift before looking up the address is fine; shipping it first is not. Correctness-only rewards score the tool name and its parameters, so a perfectly-formed call in the wrong position earns the same credit as the right one — and one misordered call cascades into every later step.
How to read it. The paper's “sending a package” example. Both rollouts make the same, well-formed calls, so a reward that only matches tool names and parameters cannot tell them apart. Scoring the order of the calls and the final state can. Bar lengths are illustrative. Schematic.
Correctness-only RLVR
Matches tool names and parameter values against the ground truth. Execution order, failed calls and the resulting system state are invisible to the reward, so the model fits what a call looks like rather than how to reach the goal.
Procedure-aware RL
Scores the invocation chain itself: graded credit for partial ordering, a check that the trajectory ends in the correct state, and an explicit penalty for structural call errors — tool calling becomes a trainable procedural skill.
Method
The policy is fine-tuned with GRPO (KL-regularized) on a weighted sum of four reward terms. Only the reward changes — the RL algorithm stays standard.
How to read it. Correctness-only RL stops at stage 2. Procedure-Aware RL adds the procedural reward (NDCG over the invocation order plus a final-state check) and an error-aware penalty for malformed calls; only the reward changes, the RL algorithm stays standard. Schematic.
The procedural term uses normalized discounted cumulative gain so that partially-correct orderings are ranked above invalid ones, instead of collapsing to a binary match. The state check gives credit whenever the agent eventually reaches the correct final state, which keeps reasonable detours — listing a directory before creating a file, or recovering from a failed call — from being punished as mistakes.
How to read it. Numbers are each call’s position in the expected order. A binary match is all-or-nothing, so a nearly right order scores the same as an invalid one; the procedural term scores order with normalized discounted cumulative gain, which ranks partially-correct orderings above invalid ones. Its final-state check credits a rollout that reaches the correct state, so a reasonable detour is not punished. Bar lengths are illustrative. Schematic.
Results
+2.6
BFCL-V3 avg. acc over ToolRL (3B)
+5.7
API-Bank avg. acc over ToolRL
+8.4
Bamboogle accuracy over ToolRL
3B
matches far larger proprietary models
Figure 1. Qwen2.5-3B-Instruct trained with procedure-aware rewards, against the same base model and ToolRL's released correctness-only checkpoint, on three tool-use benchmarks.
Table 1. BFCL-V3 accuracy, averaged over its five sub-categories (including relevance and irrelevance detection), and on multi-turn tasks. The base model, ToolRL and Procedure-Aware RL share the Qwen2.5-3B backbone; the other five models are references.
Model
BFCL-V3 avg. acc
Multi-turn acc
Qwen2.5-3B-Instruct (base)
57.31%
1.00%
ToolRL (correctness-only RL, 3B)
63.07%
3.88%
GPT-5-mini
57.11%
5.50%
GPT-4.1-mini
64.65%
2.50%
Gemma-3-12b-it
64.69%
5.75%
Gemini-2.5-Flash-Lite
57.94%
7.63%
Claude-Haiku-4.5
47.26%
1.75%
Procedure-Aware RL (ours, 3B)
65.67% Best
7.25%
Proprietary and larger open models are evaluated under BFCL's official per-model protocols, so they are a reference point rather than a matched comparison. The matched comparison is the base model and ToolRL, which share the Qwen2.5 backbone.
Both new terms are load-bearing
Figure 2. Removing either term costs more than two points, and both together account for the whole gap to correctness-only RL. Scores here average the three BFCL sub-categories (non-live, live, multi-turn), so they are not directly comparable with the table above.
Dropping the error-aware penalty mostly hurts single-turn accuracy, where invocation errors go unpunished; dropping the procedural reward mostly hurts the multi-turn split, where ordering is what matters. All differences are significant under a paired bootstrap on per-sample outcomes.
What the model learns
Rewarding sequence quality buys behavior nobody wrote into the reward. On a multi-turn file-system task the trained agent navigates before it creates — cd first, then the file — while the correctness-only baseline recreates a directory that already exists and then cascades through the resulting state errors. The same training also improves recovery: after a failed call the agent retries along a corrected path instead of repeating the failure.
How to read it. Neither behavior is written into the reward; both come from scoring the sequence. On a multi-turn file-system task the trained agent navigates before it creates (cd first, then the file), while the correctness-only baseline recreates a directory that already exists and cascades through the resulting state errors. After a failed call, the agent retries along a corrected path. mkdir, cd and touch are the calls from the paper’s Table 4 (arguments abbreviated); the recovery-row calls are generic. Schematic.
BibTeX
@inproceedings{zheng2026procedure,
title = {Procedure-Aware Reinforcement Learning for Tool-Augmented
Large Language Models},
author = {Qinglong Zheng and Jiajun Fan and Chaoran Cheng and Ge Liu},
booktitle = {Conference on Language Modeling (COLM)},
year = {2026},
url = {https://openreview.net/forum?id=d4wrBuJ4xo}
}
WATCH THE FILM