Artificial Analysis launches a unified metric for comparing search providers in a sandbox environment

The Artificial Analysis launches the Artificial Analysis Search Index, a weighted metric that ranks Search API providers by the answer quality they enable an agent to produce rather than by speed or index coverage alone. The index aggregates an equally weighted average of three separate tests. **DeepSearchQA** contains 900 broad research questions whose answers are lists of items, scored with F1 against a reference list; a representative example asks to enumerate all whispers that enable teleportation to Wilderness in Old School RuneScape and require more than four runes. **AA-Omniscience** adds 600 private samples, balanced 100 per domain across six domains, and contributes only accuracy to the metric, not its full index. **BrowseComp** contributes a hard sub-set of 200 samples drawn from 1,266. All three are evaluated by **GPT-5.6 Luna (medium)** with dedicated rubrics.

To ensure a fair comparison, all parameters are locked: the answer model is GPT-5.6 Luna (medium) at medium reasoning effort, temperature 0.6, output ceiling of 127 999 tokens, a budget of 25 agent steps with unlimited tool calls, up to 10 search results per query, and two tools, **web_search** which sends the query to the tested provider’s API and returns the raw response, and **web_fetch** which downloads textual content from a found address. The extraction layer is text-only with a 15-second page timeout, and the format remains naïve to the provider, the model sees the original JSON. Contamination filtering is enabled by default.

The fixed agent loop is managed by **Stirrup**, an open orchestration of Artificial Analysis. It supplies the model with the two tools, allows it to call **finish** when it deems enough material has been gathered, and returns reconstruction errors (timeouts, HTTP errors, address not reached from search) as failed tool calls that remain in the published results. Fatal search-provider errors (429, 5xx, timeouts) are retried until success and are not published. A **model_only** baseline—a single call to the model without a search tool—estimates the model’s internal knowledge and enables calculation of the true search-induced uplift.

The report splits total cost into two: the cost of the answer model (all tokens, input, reasoning, output) and the cost of the search provider itself. Artificial Analysis invites additional Search API providers to contact them to join the leaderboard; the mechanism is built so that any new provider enters the same pipeline exactly, without the need for prompt or parameter adjustments by the evaluators.

To date, search-provider comparisons have been based mainly on latency, coverage and price per call. The new metric forces the market to measure **final answer quality** in a realistic agent scenario that searches, reads, searches again and summarizes—the exact flow that RAG and automation products run in production. With model and procedure fixed, the difference between a provider returning ten relevant results and one returning ten noisy results translates directly into a point-percentage gap in F1 or accuracy, a number that can be monetized.