Linkup releases SPARSEUP, a 149M-parameter sparse encoder under Apache 2.0

Linkup's research team has open-sourced SPARSEUP, a sparse embedding model built on a 149-million-parameter ModernBERT backbone. The model is available on Hugging Face under an Apache 2.0 license and loads through Transformers or Sentence Transformers with trust_remote_code=True. According to the model card, SPARSEUP achieves an average 56.4 nDCG@10 on BEIR-13, the highest result Linkup is aware of for a public-vocabulary sparse encoder below 150 million parameters.

Most open retrieval models today are dense: a single vector per text. Sparse models instead emit weights across the vocabulary, each dimension corresponding to a real token, which lets them be indexed in inverted indexes and inspected manually. They also tend to handle rare words better. The catalyst for SPARSEUP was LightOn's release of open data, a training recipe, a dense model (DenseOn) and a late-interaction model (LateOn). SPARSEUP fills the missing sparse slot using the same backbone family and fine-tuning data, enabling a direct three-way comparison of retrieval styles.

Training starts from the LateOn-unsupervised checkpoint, which lacked an MLM head, so the team grafted back ModernBERT's original MLM head. Fine-tuning used LightOn's data mixture with contrastive learning only: 7 hard negatives per query sampled from a pool of 50, plus in-batch negatives. No cross-encoder distillation was used, and the entire training run fit on a single H100. Vanilla SPLADE on this backbone produced massive "bags" full of stopwords. Linkup fixed this with three changes. First, logit shifting: the encoder computes log(1 + ReLU(x - 15)) because ModernBERT's logits sat too high and saturated the log at initialization. Second, top-k per position: each input token retains only the 12 strongest vocabulary dimensions before max pooling, limiting per-token expansion rather than total vector size. Third, case folding: byte-level BPE treats heat, Heat, Ġheat and ĠHeat as separate IDs; SPARSEUP folds them to a single ID keeping the largest weight, reducing output dimensions from roughly 50 thousand to roughly 34 thousand. Queries and documents receive [Q] and [D] prefixes, and scoring is a dot product. Maximum lengths at evaluation are 128 tokens for queries and 512 for documents.

Against other sparse encoders on BEIR-13 (nDCG@10, excluding MS MARCO), SPARSEUP leads according to the model card. The controlled comparison is less flattering: with backbone and data held constant, LateOn scores 58.9, DenseOn 57.9, and SPARSEUP 56.4. Notably, SPARSEUP uses approximate Seismic search while LightOn reports exact search. SPARSEUP wins on ArguAna and Touché and beats DenseOn on HotpotQA, but trails on more semantic datasets, with the largest gap on FiQA; DBPedia is another weak spot. On decontaminated BEIR, the gap to DenseOn narrows to 0.17 points. Linkup cautions that decontaminated NQ and MS MARCO contain only 21 and 46 queries respectively, making those results noisy.

On MS MARCO, SPARSEUP produces an average of 47 non-zero elements per query and 190 per document. For comparison, SPLADE-v3 yields 25 and 170 respectively. With a Seismic inverted index, the model reaches over 97% recall versus exact search at roughly 380 microseconds per query, single-threaded. Linkup says inflating vector size could have added 1 to 2 BEIR points, but they chose to remain sparse.