HumanEvals library launches, adding live human judgment to model evaluation loops

An open-source library called HumanEvals lets model developers embed live human judgment into their automated evaluation pipelines without leaving the notebook or building a response-collection infrastructure from scratch. The library, built by the Datapoint team, wraps the API of their labeling platform and returns Score objects compatible with autoevals, so existing code that runs automatic benchmarks can work with human annotators without modification.

the engine behind the scenes: 5,000 tags per minute

The real power resides in Datapoint itself: the same pool of annotators that labs leading image, audio and video models use to compare checkpoints against competitors. According to the company, the capacity exceeds 5,000 tags per minute, and the library supports four evaluation modes—pairwise comparison, fixed-scale rating, multiple-choice (including classification against an expected answer), and relative ranking of several candidates. Each mode accepts a free-form prompt and input items—text, local media files that upload automatically, or structured choice/rating objects—and returns a score in the [0, 1] range with full metadata: raw vote counts, agreement metrics, weighted confidence versions, and a job_id for tracking.

free sandbox for development, payment only in production

Every scorer can set the parameter `sandbox=True` to run the task on Datapoint’s test pool, using zero credits, test annotators, and an identical API mechanism. This lets users wire the end-to-end pipeline before paying for real labeling. In batch calls (`eval_batch`) a single Job is created for all items, and annotators work in parallel: a broad crowd finishes in minutes, a narrow target may take hours. For long-running tasks the library exposes `submit()` which returns a job_id, then `EvalJob.attach()` which allows later attachment—from another process—to check live progress, peek at partial results with `wait=False`, or cancel and receive a refund for unused reservation.

auto-evals compatibility and swapping LLM judges

The HumanComparison interface also supports a single-call autoevals style, `scorer(output=…, expected=…, input=…)`, which returns the probability that a human would prefer the output, directly comparable to autoevals Battle. This opens the door to calibrating LLM judges against ground-truth human data, building golden consensus datasets, and gathering preferences for RLHF with per-annotator responses rather than only averages. The only runtime dependency is `httpx`, and the library requires Python 3.10 or newer.

what it means in practice for research teams

Until now, adding human judgment to a training pipeline required building an internal dashboard, recruiting annotators, managing queues and computing agreements, or buying a managed service without control over the logic. HumanEvals pushes all of that into a single library that speaks the same language as automatic benchmarks. The result: researchers can run nightly human evaluation on any checkpoint, pit it head-to-head against a competing model, and obtain a data-driven release decision without leaving the CLI.