340M-parameter GLiNER2.5-Decide runs on phone GPU with numerical parity to reference model

Fastino researchers have released a LiteRT build of GLiNER2.5-Decide, a DeBERTa-v3-large derivative that performs zero-shot classification with arbitrary labels at inference time. The fp16 conversion preserves full numerical identity with the fp32 reference: 361 of 361 test requests returned identical decisions on desktop, and 126 of 126 (request, window) pairs matched on a Galaxy S26. This is not aggressive quantization that sacrifices accuracy. The fully connected layer weights — 146 tensors — remain in fp16 with dequantization to fp32 at runtime, while activations and other constants stay in fp32.

Single-pass architecture for multiple tasks

The model accepts text and a task dictionary — intent, urgency, sentiment, multi-label topics — and returns a decision for every task in a single forward pass. The host handles word embeddings before the graph and logits-to-decisions conversion after, governed by an explicit contract that defines the encoded token sequence, marker positions, and decision rules. Three graphs with different window sizes (s128, s256, and a third) are supplied; the developer selects the smallest that accommodates the schemas and input text, subject to a 32-label ceiling. The total package — three graphs, the fp16 embedding table, and the tokenizer — weighs roughly 2.45 GB.

Performance on real hardware, not simulation

On a Galaxy S26 (model SM-S942Q, SM8850 processor, Android 16) running LiteRT 2.2.0 with explicit GPU acceleration in FP32, inference latency ranges from 66 to 70 milliseconds for a 128-token window and approximately 175 milliseconds for a 256-token window. Desktop CPU times with 4 threads are comparable. The acceleration is explicitly configured as GPU FP32, not the GPU's native fp16, which accounts for the numerical stability. Other Android GPU families were not tested in this release, a limitation the developers note explicitly.

Practical implications and deployment caveats

The ability to run a 340M-parameter model with a dynamic label schema on an edge device opens the door to intent routing, urgency detection, and topic classification without a server, without network latency, and without sending user text to the cloud. The example code (Python and Android) pins gliner2 2.0.0 as the locked schema and tokenizer, eliminating the need for a separate checkpoint download. On the other hand, 2.45 GB remains a heavy payload for a typical application, and execution requires LiteRT 2.2.0 with GPU FP32 support — a non-trivial dependency for broad deployment.