Keenable launches NEEDLE: live search benchmark that rebuilds query set every hour

The problem with static benchmarks is that they give a search agent easy access to a public answer repository, turning the dataset into a key-answer store. The model can drop the labels midway through evaluation and skip the retrieval step. The same issue appears when knowledge is already encoded in the model’s parameter memory: a correct answer does not prove that search worked. Keenable’s solution is NEEDLE, an open-source evaluation framework that generates fresh queries from public sources instead of freezing a fixed set.

NEEDLE operates across five “verticals,” updating continuously. The acronym stands for News, Everyday, Expert, Deep-tail, Legal Evaluation. Each vertical simulates a different agent intent. The News vertical pulls a current item from 124 RSS feeds and Google Trends each hour and translates it into a keyword query. The Everyday (Finance) vertical extracts registration facts from Wikidata and GLEIF and quarterly 10-Q data from SEC XBRL daily. The Scholar vertical turns a single paper into four query styles—reduced title, full-text excerpt, natural-language hint, and “on the tip of the tongue” description—from arXiv and Europe PMC. The Deep-tail vertical samples rare queries from public agent repositories such as DeepResearchGym, OpenResearcher and LRAT. The Legal Evaluation vertical retrieves recent judgments from CourtListener covering 14 federal courts and sections of the eCFR.

Scoring methods differ per task. For News and Deep-tail, which lack a single correct answer, a language model grades each result on a 0-4 scale, and nDCG@5 is reported with a penalty for duplicate URLs. Finance reports answer-recall@5: whether the fact appears within the top five passages. Scholar and Legal are known-item tasks measured by identifier match.

All engines receive exactly the same query text, one call at a time, so latency percentages are comparable and no engine sees parallel load. Judgments are based on the engine’s ranking, titles and snippets; pages are never fetched, results are never re-ranked, evidence is limited to 2,000 characters for all, and the judge does not see the engine name. The more interesting ceiling is the “ultimate oracle.” For each query, NEEDLE aggregates the results returned by all engines into a synthetic oracle, sorts the union by relevance, and creates an empirical upper bound based on what the market collectively retrieved. The gap to the ultimate reflects an upper bound on current agent search quality. A large gap means better results existed but no engine surfaced or ranked them well; a weak ultimate score indicates that even after pooling all providers, the benchmark found few strong evidences, i.e., NEEDLE can distinguish between a ranking problem and a retrieval problem that affects the whole market.

The numbers below are seven-day averages for the window ending 28 August 2026. In Finance, the picture is close to the ceiling: Exa 0.910, Keenable 0.872, Perplexity 0.871, Google 0.847, versus an ultimate ceiling of 0.965. In Scholar, the spread is wide: Keenable 0.774 versus Tavily 0.310 versus a ceiling of 0.869, because title-only queries are answered from metadata while full-text queries are not. Deep-tail data were omitted in the source.

NEEDLE is released as an evaluation harness, not a product. It is a Python CLI installed with uv sync and invoked with two sub-commands for each benchmark, generate and run. An OpenRouter key is required for judging and an API key for each engine under test. It runs on a laptop or in CI, and the code lets users reproduce all query streams in use as well as the ranking quality judgments.