Google's ToolGrad flips tool-call data generation, hits 99.8% success rate

A research team from Google, the University of Tokyo, RIKEN AIP and Tohoku University has published ToolGrad, a method that inverts the standard pipeline for generating tool-calling training data. Instead of starting from a user query and searching for a tool chain that satisfies it, ToolGrad first constructs a verified chain by executing APIs in practice, then attaches a matching query. Gemma-3 models fine-tuned on just 500 ToolGrad examples — ToolGrad-500 — reach scores on par with leading proprietary models on the Berkeley Function Calling Leaderboard.

Previous pipelines such as ToolBench and ToolACE operated in the opposite direction: they sampled an API pool, asked an LLM to invent a plausible user instruction, then ran a depth-first search agent to find a tool path that fulfilled it. The search offers no guarantee of success; when it hits a dead end, the compute spent exploring is wasted and the example is discarded. The paper characterizes the process as distilling expensive trajectories from a complex agent that often fails — an inherent inefficiency.

ToolGrad reverses the order. Each iteration runs four modules in sequence: an API Proposer narrows a sampled set of APIs to individual candidates that can extend the workflow; API Executors run the candidates in parallel and return detailed execution reports; an API Selector examines the reports, picks the most successful call and appends it to the chain, using its textual feedback as a guiding gradient; an LLM Updater rewrites the synthetic query and the model response to align with the updated API set. The default configuration runs 10 iterations on 50 sampled APIs per workflow, producing one complete example: a query, a verified tool chain, and a final answer.

The researchers evaluated data generation on the ToolBench repository, which contains more than 16,000 real APIs, comparing ToolGrad against the original DFS approach. According to the paper, the pass rate jumped from 63.8% to 99.8%; verified calls per example rose from 2.1 to 3.4, indicating longer chains; tool-use steps fell from 34.3 to 20.0; and LLM calls per example dipped slightly from 64.5 to 63.9. The remaining 0.2% failed when the agent could not obtain a successful response from three selected APIs across all 10 iterations, and an empty example was retained.

To create ToolGrad-500 the researchers used Gemini 2.5 Flash-Lite, then applied the data for post-training of Gemma-3 at 1B, 4B and 12B parameters. Evaluation was performed on the Berkeley Function Calling Leaderboard, which uses a different tool set from ToolBench — an out-of-distribution test with unseen tools. Fine-tuning on ToolGrad-500 improved tool-calling scores across all sizes. ToolGrad-12B scored 83.1, versus Gemini 2.5 Pro at 83.2, Claude 4.5 Opus at 82.8, and GPT-5 at 74.4 as measured at publication time. The 12B student surpassed the teacher model that generated its training data, and led open-source tool-calling specialists such as ToolACE and Hammer-2.1-7B.

The code is released under Apache-2.0. The ToolGrad-500 dataset and the three models (1B, 4B, 12B) are available on Hugging Face, and a PyPI package enables quick installation. Reproduction scripts in the repository target BFCL V1 and V2 via an adapted fork and run inference inside a vLLM Docker container, allowing the full pipeline to execute without external dependencies.