My RAG pipeline could not find an error code

The question that broke it was boring: someone asked about a specific error code. The document containing that error code was indexed, chunked and embedded. The pipeline returned three passages about error handling in general and not the one passage that contained the string.

That is not a bug in the sense of something being broken. Every component did exactly what it was built to do. It is a design that was wrong about what people ask.

Embeddings are about meaning, and an error code has none

A vector search asks what is this passage about. That is the right question for most natural language and the wrong question for an identifier. A model name, an error code, a configuration key, a tenant identifier — these carry almost no semantic content. Two passages mentioning entirely different error codes look nearly identical to an embedding model, because the meaning of both is roughly error happened here.

A keyword index is excellent at exactly this and hopeless at paraphrase. So the answer was not to pick the better one. It was to run both and fuse the rankings — reciprocal rank fusion over a vector search and a text index, so a passage that either method ranks highly survives.

Retrieval quality stopped being a single number I could tune and became a question of coverage: which class of question does each retriever fail at, and does the other one catch it.

The chunker was splitting code in half

The original version cut every document into fixed five-hundred-word windows. Simple, predictable, and indifferent to what it was cutting — which meant code fences split across two chunks, so neither half was valid and neither was useful. Sentences were cut mid-clause for the same reason.

Chunking is now structure-aware. It splits on headings, paragraphs and fenced blocks, targets three hundred to seven hundred tokens with about fifteen percent overlap, and treats a code fence as atomic on purpose. An oversized chunk is worse than an average one; a broken code block is worse than both.

MIN_CHUNK_TOKENS = 300
MAX_CHUNK_TOKENS = 700
OVERLAP_RATIO    = 0.15

// A fence is never split. A paragraph that exceeds the budget
// is split at sentence boundaries; a "sentence" with no
// terminal punctuation is hard-sliced rather than left whole.

That last case sounds theoretical until a document contains one. Then it is a single atom larger than the context window, and the pipeline stops.

Two hundred round-trips for one document

Embedding ran one HTTP call per chunk. A two-hundred-page document therefore meant roughly two hundred sequential requests, and the pipeline spent nearly all of its time waiting rather than computing.

The inference server had accepted a batched input array the whole time. Thirty-two chunks per call turned the dominant cost from network latency into actual work. No cleverness, just reading the interface properly — which is where a surprising share of performance lives.

An answer without a citation is a rumour

The first version stored raw float arrays and the chunk text. Nothing else. So when the system answered a question there was no way to say where the answer came from, and no way for a reader to check it.

Every chunk now carries the heading it sits under, the page it came from, and its character offsets in the original. The answer arrives with somewhere to look, which changes what the feature is for: not a system that tells you things, but one that finds you the paragraph and lets you decide.

Given what models do with facts, that distinction is doing a lot of work.