H Company launches NeoMME: single bidirectional encoders that drop the vision tower and causal decoder

Most visual document retrievers in production today are legacy systems. ColPali and the models that followed take a generative vision-language model and repurpose it as an encoder; the result still drags along a vision tower that was pretrained separately and a causal decoder that never generates a token. That is parameter and compute overhead for a task that only needs representations. H Company, a French startup operating from Paris, is now releasing NeoMME, a family of bidirectional encoders in 260 million and 800 million parameter sizes that discard both components. A single transformer processes multilingual text tokens and raw 32×32 RGB patches through the same layers, from fully random initialization. Text enters via an ALBERT-style factorized embedding: a 256-dimensional lookup projected to model width. Images are cropped into non-overlapping patches and projected by a two-layer MLP trained from scratch, with no patch-merging module and no SigLIP2 tower.

Pretraining uses discrete masked diffusion on text, with an option to condition on visible image patches. Text-only segments draw a uniform corruption rate between 0 and 1. Multimodal segments draw from 0.30 to 1, removing the language-only shortcut and forcing the model to read the page. A cross-modal ablation confirms the effect: at 90 percent masking, visible page patches raise masked-token accuracy by 38.4 points for the 260M model and 40.5 points for the 800M model. Each run processed roughly 524 billion compressed input tokens, of which about 290 billion were text-only, on 16 and 32 H100 accelerators respectively. The tokenizer is a whitespace-free BPE with a 131,072-entry vocabulary, trained from scratch. On 14 target languages in the FLORES-200 devtest set, it emits 44.4 percent fewer tokens than ModernBERT.

NeoMME-Retriever adds two heads trained jointly on the shared backbone: a dense head with matryoshka widths, and a late-interaction head that projects each token and patch to 128 dimensions. A single forward pass returns both. On ViDoRe v3, the 260M model reaches 0.523 nDCG@10 and the 800M model reaches 0.556. The 260M result sits just 0.002 behind ColQwen2.5-v0.2 at 3.75 billion parameters, and 26.1 points above the best other model under 300M. The 800M model falls 0.9 points behind Vultron Retriever Flash at a similar size. In textual retrieval the picture is less flattering: on BEIR-15, late interaction scores 0.4881 and 0.5126, versus 0.5722 for LateOn at 149M parameters. The authors attribute this partly to supervision scale: NeoMME saw roughly 430 thousand text-query examples, compared with about 660 million contrastive examples for mLateOn.

Late-interaction indexes are expensive. A 2048×2048 page yields 4,162 vectors, roughly 1.5 megabytes per ViDoRe v3 document in float32. Two methods bring this down. Hierarchical token pooling at a factor of 10 with int8 queries and documents gives 39 kilobytes per page, a 39.4× reduction while retaining 99.16 percent of the baseline nDCG@10. A pooling factor of 8 with int8 queries and binary documents gives 6 kilobytes, a 255.5× reduction while retaining 95.19 percent. Indexing is fast: the small model processes 51.3 pages per second on a single L40S, and query encoding on a CPU-only host takes 78.3 milliseconds. Every checkpoint is released under Apache 2.0 with day-zero Hugging Face Transformers support.