Harold launches with the most comprehensive LLM cost-performance database to date

A new comparison platform called Harold went live on 17 September 2026 with what appears to be the largest public dataset for evaluating language-model cost-effectiveness: 100 benchmarks, 840 models and 16,429 measured results. Unlike tables that stop at price per million tokens, Harold shows the real cost per task — a metric that captures what it actually costs to run a full workload, including overhead, retries and performance differences between inference providers.

Behind the numbers

The standout figure is coverage. The hundred benchmarks span coding (HumanEval, MBPP), mathematical reasoning (GSM8K, MATH), natural-language and multilingual tasks. Every model is tested against every benchmark where it has public results, and those results are cross-referenced with current inference pricing from providers that publish rates — not list prices, but prices measured in practice. The output is 16,000 rows that can be filtered by task, budget, hosting region and evidence requirements.

Benchmaxxing score and cost per task

Harold's core innovation is a "Benchmaxxing" score, described by its creators as the first attempt to quantify the cost-performance ratio in a way that allows direct comparison across models of different sizes and architectures. Instead of looking at a raw score on a single benchmark, the weighted score combines performance across multiple tests relevant to a given task and divides by the computed task cost. Alongside it sits a "Composite Score" that aggregates several benchmarks into broader task families such as "general coding" or "mathematical reasoning."

EU hosting and evidence requirements

A dedicated section of the interface handles "EU hosting," a strict technical definition: inference counts as EU-hosted only if it runs in a European region of the provider — AWS Bedrock EU cross-region, Azure Europe Data Zone, or a provider whose entire public fleet is documented as hosted in the EU. A European billing region, an EU-registered company or an EU control plane are not sufficient on their own. One exception based on a declared company policy remains in the table and is labelled "EU equivalent." Above every pricing table the relevant evidence requirements are listed: benchmark evidence, a priced provider, and tokens measured for the task.

What this changes in practice

For developers choosing a model for production, the difference is an apples-to-apples comparison: not "what does GPT-4o cost per million tokens" but "what will it cost to run my test suite on GPT-4o versus Claude 3.5 Sonnet versus Llama 3.1 405B at provider X," with performance data from the exact same benchmarks. The interface lets users filter by maximum budget per task, geographic hosting requirement and required evidence level, eliminating the manual round-trip between pricing pages, benchmark reports and provider documentation.

Pieces still missing

Harold does not yet provide latency metrics, uptime SLAs or provider rate limits — parameters that can decide a production deployment as much as cost per task. There is also no public API for pulling the data into CI/CD pipelines, although the creators hint that one is on the roadmap. The bottom line: Harold closes a significant information gap in model selection, but it is a decision-support tool, not a substitute for real load testing on your own system.