Fastino Labs releases 340M-parameter decision model that runs on CPU and returns structured answers

with confidence scores Fastino Labs has released GLiNER2.5-Decide, a 340-million-parameter open-weight decision model that accepts text and a schema of typical questions and returns structured answers containing probability distributions, confidence scores, and constraint-feasibility metadata. The model targets the frequent decision points in agent pipelines: routing, triage, tool selection, and guardrails. Weights are distributed under Apache 2.0, installable via `pip install gliner2`, and run on CPU, GPU, or in air-gapped environments; the company also offers managed inference and fine-tuning through the GLiNER API.

What it actually is

GLiNER2.5-Decide is not a generative model but a classifier built on a DeBERTa-v3-large encoder and fine-tuned from gliner2-large-v1. It does not generate tokens and does not require a prompt template. Label sets are passed at inference time: each question in the schema declares the allowed answers, whether a single, multiple, or ranked response is required, and can carry instructions, examples, label descriptions, and rules linking answers across questions. The two-stage pipeline has the encoder read text and schema together and score every allowed answer, while a constrained decoder searches for the highest-scoring joint assignment the rules permit. Fastino stresses explicitly: the model does not reason, does not explain, and does not answer open-ended questions; it is a specialist for operational decisions.

Joint decoding and why it matters

The team demonstrates the value with a guardrail example. Under independent decoding the model flagged a prompt injection at 0.82 while simultaneously classifying the same prompt as safe at 0.52 — the attack was detected but the two outputs contradicted each other. Joint decoding applies a rule that any detected harm forces an "unsafe" verdict, and the model returns safety=unsafe and harm_type=prompt_injection together. Downstream code can use those scores to block, route, or escalate. Schemas support implications, exclusivity, cardinality limits, and ordinal bounds; the same encoder can also extract entities, relations, and structured records with character-level offsets in a single forward pass (classification answers do not return evidence spans).

Performance and benchmarks

Evaluation was conducted on Fast Decisions, an internal held-out suite of 5,100 test examples across 17 datasets covering customer operations, domain routing (banking, clinical, travel, benefits), and general content understanding. The metric is exact-match accuracy: a prediction counts only if its label set is identical to the reference. GLiNER2.5-Decide led on 9 of the 17 datasets. Intent routing was the strongest domain: 75.3% on support intents and 64.3% on banking intents, gaps of 18.6 and 8.6 percentage points over the next-best models. The benchmark is internal and has not been peer-reviewed, so the numbers reflect the vendor's measurement only.

Latency on real hardware

The team measured the checkpoint end-to-end at batch size 1 with a two-head schema and 15 labels. At 64 tokens, median (p50) latency was 167.3 milliseconds on 48 vCPU Intel Xeon Platinum 8581C, versus 43.6 ms on NVIDIA T4, 43.4 ms on L4, 38.3 ms on V100, and 47.3 ms on A100. On short requests the fixed overhead of preprocessing and kernel launch dominates, so the GPUs remain within 9 ms of each other. At 1,024 tokens the A100 takes the lead with 52.6 ms, against 75.6 ms on V100 and 131.4 ms on L4.

Availability and usage

The multi-head call is demonstrated in the model card: load `AutoExtractor.from_pretrained("fastino/GLiNER2.5-Decide")` then call `classify_text` with text and a heads dictionary — for example intent, urgency, route — where each head defines its allowed labels. Single-label heads return a string; multi-label heads return a list. The open weights (open-weight, not fully open-source) enable local embedding without cloud dependency, which is exactly the kind of tool MLOps teams look for when they need deterministic, fast, auditable decisions inside the agent loop.