RAG moves beyond vector search as architecture shifts

Until recently, almost all Retrieval-Augmented Generation (RAG) systems were built the same way: documents were split into chunks, turned into embeddings (vector representations of text), stored in a vector database, and queried for context for the LLM. That was a marked improvement over ordinary keyword search, but as corporate knowledge bases grew, pure semantic search proved insufficient.

The gap originates in the type of queries. When a user searches for a specific function, error code, API method, or a document from a particular version, embedding-based search does not capture the information with the required precision. The situation becomes even more complicated when the query concerns information that links several different entities. In such cases, spatial proximity between vectors simply does not provide an answer. As a result, modern AI systems are moving to a hybrid retrieval model in which several algorithms operate in parallel, each handling a different sub-task.

Instead of relying on a single algorithm, the new architecture adds extra layers. BM25 keyword search is integrated to find more exact matches. A Metadata Filtering mechanism limits the search by document version, date, author or project. A reranker receives the search results and reorders them to keep only the most relevant documents for the model. At a higher level, a Knowledge Graph enables the system to locate not just documents but also relationships between entities such as users, services and events. Tools like Graphiti already combine a knowledge graph, semantic search and additional retrieval mechanisms rather than building a system solely around vectors.

Anthropic is also discarding the classic RAG pattern. The team behind Claude Code says the engineers omitted pre-forming context with a traditional RAG pipeline. Instead, the agent performs on-the-fly searches and incrementally gathers the needed information while executing the task. This approach is especially cost-effective for large codebases and constantly changing contexts, allowing the agent to handle complexity that a static retrieval system cannot resolve.