Papers with Code builds hybrid search on Hugging Face

Papers with Code disclosed the architecture behind its search engine, a hybrid system that combines full-text lexical search in PostgreSQL with semantic retrieval via pgvector, merging results with the Reciprocal Rank Fusion (RRF) algorithm. The engine currently serves more than 110,000 papers sourced from arXiv and Daily Papers and is designed to handle navigational queries (“the original BERT paper”), partial titles, typographical errors, and situations where the embeddings service is cold, overloaded, or unavailable.

The team deliberately split the workload: the expensive corpus construction, which requires high throughput, runs as Hugging Face Jobs on burstable GPU compute on demand. The resulting vectors and metadata are stored in Hugging Face Storage Buckets, acting as a durable staging layer between the database, experiments, and jobs. Only the live query-embedding step executes on the request path, behind a protected Hugging Face Inference Endpoint, with an immediate fallback to full-text search if the endpoint does not respond in time. This separation provides both semantic power and low latency.

Embedding format is treated as a versioned API to avoid silent drift. Each paper is encoded as a normalized title, two newline characters, and a normalized abstract. For every vector generation the system records the model repository and exact revision, output dimension, input-format version, a flag indicating whether the input is a query or a document, the normalization method, and a content hash of the title and abstract. This metadata enables full reproducibility and prevents mismatches when model versions, prompts, or abstracts change.

The chosen model is Qwen/Qwen3-Embedding-0.6B locked to a specific revision, producing L2-normalized 256-dimensional vectors. The selection was based on the MTEB benchmark, the standard for comparing embedding models. The Qwen3 generation adds two relevant capabilities: dynamic embedding size support through Matryoshka Representation Learning (MRL), which allows a trade-off between quality, speed, and storage, and the ability to pass an instruction prompt that steers the model toward the task. The team selected 256 dimensions to keep search fast and to accommodate the instruction prompt.

The main lesson from production is that graceful degradation is not optional. Separating the costly offline path from the hot online path is a reliability requirement: when the embeddings endpoint is unavailable, the system does not return an error but falls back to full-text lexical search, still returning accurate titles and arXiv identifiers. Simultaneously, versioned input format and content hashes prevent the classic problem of stale vectors persisting after abstract updates or model changes. The result is a search engine that understands semantics but does not stall when the infrastructure hiccups.