(Half A and Half B) we constructed the upgraded pipeline and watched it work: pointed at a analysis paper, a NIST normal, and a report with a damaged desk of contents, it returned a typed, cited reply every time. So why does it work? And would a naive RAG, the sort nearly everybody builds first, embed the pages, hold the closest few, ask, have executed the identical?
This text runs the 2 aspect by aspect. The quick reply isn’t any, and the helpful half is the place the naive one breaks: not in a single place, however at every of the 4 bricks in flip, doc parsing, query parsing, retrieval, and era, every time for a purpose you’ll be able to title and repair.
It is usually why the same old reflexes — rewrite the immediate, shrink the chunks, swap the embedding mannequin — don’t assist. The improper reply doesn’t come from the immediate. It comes from a brick upstream handing the mannequin the improper context, which the mannequin then solutions faithfully. Getting all 4 bricks proper, not simply the immediate, is what this collection calls context engineering. Each failure beneath is an actual run, by no means an assumed one; the companion pocket book reproduces every one. Article 7quinquies (most RAG hallucinations are retrieval failures) makes the identical case on the retrieval stage on one doc; this one exhibits it throughout all 4 bricks.
📓 The runnable pocket book for this text is on GitHub: doc-intel/notebooks-vol1. It runs the naive baseline and the upgraded pdf_qa aspect by aspect on every doc beneath, prints each solutions with their confidence, and reproduces each cross within the figures by yourself machine.
1. The naive baseline, brick by brick
The baseline is the 100-line pipeline from Article 1 (minimal RAG): parse the PDF, flip the query into key phrases, hold the highest few pages by match, ask the mannequin to reply. On a brief, clear, prose paper it really works, and Article 9’s personal checks confirmed it: on Consideration Is All You Want and the RAG paper, naive and upgraded each reply appropriately.
The upgraded pipeline is the similar 4 bricks, every with a tighter contract: parsing returns a relational line_df as a substitute of flat textual content, query parsing expands the question into the doc’s vocabulary, retrieval routes on the desk of contents, era returns a typed reply. A naive RAG runs the unfastened model of each brick. The hole opens precisely the place every unfastened brick fingers the mannequin the improper factor. Every part beneath isolates one brick: it swaps that brick for its naive model and leaves the others cheap, so the failure, and the repair, belong to the brick named within the heading.
Learn it high to backside: the identical symptom, a assured improper reply, arrives 4 alternative ways, and a special brick closes every. The remainder of the article walks the 4, every on an actual doc.
2. Parsing: a desk flattened into noise
Doc: World Financial institution Commodity Markets Outlook (a report constructed round value tables). Query: What’s the 2025 annual common value forecast for U.S. pure gasoline (Henry Hub)?
A naive pipeline parses a PDF the frequent method: dump flat textual content with get_text() and minimize it into fixed-size chunks. That’s wonderful for prose and deadly for tables. A value desk is a grid; flat extraction linearises it, and a fixed-size chunker then cuts the stream wherever the character price range runs out. The row label Henry Hub lands in a single chunk and its 3.5 cell in one other. Retrieval fingers the mannequin chunks that by no means carry each, and it solutions truthfully: “not acknowledged in these strains,” confidence 0.00.
Nothing was hallucinated. The quantity was within the doc; the parser destroyed the one factor that made it a solution, the alignment between the label and the cell.
The repair is relational parsing. As an alternative of flat textual content, the upgraded brick returns a line_df: one row per textual content line, every with its bounding field. The desk’s row stays intact, the mannequin reads Henry Hub and 3.5 on the identical line, and returns “$3.5 per mmbtu” at confidence 0.99. That is the entire argument of the parsing brick (Article 5): cease returning flat textual content, hold the relational form the reply lives in.
3. Query: a phrase the doc by no means makes use of
Doc: NIST SP 800-207, Zero Belief Structure. Query: What are the pillars of zero belief structure?
The doc by no means makes use of the phrase pillars. It calls them tenets. A naive pipeline searches the pages for the question’s personal phrases, pillars scores towards nothing, and the mannequin studies “the precise pillars will not be listed,” confidence 0.20. The reply is correct there, underneath a special title; the proper pages have been by no means even searched.
This isn’t a rating downside, and no larger top-k fixes it: the question phrase merely shouldn’t be within the doc. It’s a vocabulary downside, and it lives one brick earlier than retrieval, in query parsing. The consumer’s phrase and the doc’s phrase are synonyms the uncooked string can’t bridge.
The repair is query parsing. Earlier than retrieval runs, the brick normalizes and expands the question, mapping area synonyms (pillars → tenets, ideas) so retrieval searches for the phrases the doc really makes use of. It then anchors the tenets part, and era returns all seven at confidence 0.95, noting the doc’s personal time period. The identical growth lets a consumer ask a couple of canine and match a coverage that solely ever says animal, the case Article 7quinquies (most RAG hallucinations are retrieval failures) walks intimately.
4. Retrieval: the reply beneath the cutoff
Doc: NIST Cybersecurity Framework 2.0 (32 pages, native desk of contents). Query: How is a Profile outlined in CSF 2.0?
The phrase Profile is on many pages of CSF 2.0: it names a piece, recurs in prose, heads examples. Just one part defines it. Key phrase and cosine retrieval rank a web page by how related it seems to be to the time period, and each Profile web page seems to be equally related. The naive pipeline retains the highest few by frequency, none of which is the defining web page, and the mannequin studies “not outlined in these strains,” confidence 0.10. The answering web page fell beneath the top-k cutoff.
The repair is retrieval that routes on construction. The upgraded brick reads the doc’s personal desk of contents by way of a small LLM, sees the part titled CSF Profiles, and anchors retrieval there as a substitute of on uncooked frequency. Era sees the defining paragraph and returns the total definition with a citable span, confidence 0.95.
The identical routing scales the place naive retrieval will get worse. On the 400-plus-page NIST SP 800-53 management catalog, requested for the one AU-2 Occasion Logging management, naive retrieval dilutes it previous the cutoff amongst a thousand near-identical siblings and returns “NA” at 0.00; the router locates the AU household and the AU-2 entry instantly and returns the total requirement, clause by clause, at 0.98. Routing on construction doesn’t degrade because the doc grows; rating on frequency does.
5. Era: a assured reply with no self-check
Doc: World Financial institution Commodity Markets Outlook, April 2024 (its forecasts cease at 2025). Query: What’s the 2026 annual common value forecast for crude oil (Brent)?
This time each pipelines retrieve the identical, right context: the vitality pages with the value desk. The doc merely has no 2026 row. A naive pipeline’s era brick asks the mannequin for a free-text reply, and a free-text mannequin requested a query tends to reply. It grabs the closest quantity, the 2025 worth of $79, and returns “the 2026 Brent forecast is $79 per barrel”, assured, fluent, and improper. Nothing in a prose reply might have stated “this isn’t within the doc.” That is the hallucination readers report, and it occurred on the final brick, with good context in hand.
The repair is a typed era contract. The upgraded brick doesn’t ask for prose; it asks for a schema with a complete_answer_found area the mannequin should set, an proof span it should cite, and a confidence it should justify. Confronted with a price that isn’t within the strains, the mannequin can’t quietly fill the hole: it units complete_answer_found: false and returns “the 2026 forecast shouldn’t be supplied; the newest is 2025.” The era brick doesn’t make the reply smarter. It makes a lacking or partial reply seen, so a improper one can’t ship as executed. The identical complete_answer_found area guards partial solutions on the whole: every time retrieval surfaces solely a part of what the query requested, it’s the distinction between delivery a fraction as if it have been entire and flagging the hole.
6. One root, 4 doorways
The 4 failures are one factor seen 4 methods. In none of them did the mannequin invent from nothing. Every time it answered the context it was handed, faithfully, and the context was improper: a desk the parser scrambled, pages discovered for a phrase the doc by no means makes use of, the answering web page left beneath the cutoff, or a price that was by no means there and received crammed in anyway. The label hallucination factors on the mannequin; the trigger was a brick upstream.
The frequent repair shouldn’t be a greater immediate, an even bigger mannequin, or a bigger top-k. It’s context engineering, one contract per brick: hold the doc’s relational form (parsing), search within the doc’s personal vocabulary (query), route on the doc’s personal map (retrieval), and bind the reply to a typed, checkable contract (era). Get the context proper and a assured improper reply has nowhere to return from, as a result of the mannequin is now not being requested to reply the improper context.
Two trustworthy caveats. First, naive RAG shouldn’t be all the time improper: on quick, clear, prose paperwork it retains up, and the trustworthy determine in Article 9 exhibits it. The hole opens precisely the place enterprise paperwork reside: tables, home vocabulary, size, construction. Second, these 4 crosses are those that held up. We ran the naive baseline and the upgraded pipeline on many extra (doc, query) pairs, and the place naive did wonderful we are saying so relatively than manufacture a failure. Every cross above is an actual run, not a manufactured one.
7. Sources and additional studying
The failure circumstances are actual runs of the naive baseline (pdf_qa_baseline, plus a flat-chunk parser and a free-text generator for the parsing and era bricks) towards the upgraded pdf_qa (Article 9); every one is an actual run, reproducible within the companion pocket book, not an assumed outcome. The responses.parse(text_format=Schema) sample behind the typed reply makes use of OpenAI’s Structured Outputs.
Earlier within the collection:
- Doc Intelligence: collection intro. What the collection builds, brick by brick, and in what order.
- The minimal Enterprise RAG that by no means lies about its supply: PDF in, highlighted reply out. The naive four-brick baseline this text stress-tests.
- Retrieval is filtering, not search: a psychological mannequin for enterprise RAG. Retrieval reframed as filtering, the concept behind TOC routing.
- Most RAG hallucinations are retrieval failures (Article 7quinquies, hyperlink to return). The identical argument on the retrieval stage, on one doc, together with the vocabulary-mismatch case.
- A manufacturing RAG pipeline for PDFs: relational parsing, TOC retrieval, typed solutions (Article 9, hyperlink to return). The upgraded pipeline these circumstances run on.
Paperwork used (all brazenly licensed):
- World Financial institution, Commodity Markets Outlook, April 2024. The value-table parsing case (part 2) and the absent-value era case (part 5). CC BY 3.0 IGO.
- NIST, Zero Belief Structure, SP 800-207, August 2020. The vocabulary-mismatch case (part 3). US Authorities work, public area within the US.
- NIST, The NIST Cybersecurity Framework (CSF) 2.0, NIST CSWP 29, February 2024 (DOI 10.6028/NIST.CSWP.29). The buried-definition retrieval case (part 4). US Authorities work, public area within the US.
- NIST, Safety and Privateness Controls for Info Programs and Organizations, SP 800-53 Rev. 5. The diluted-control retrieval case at scale (part 4). US Authorities work, public area within the US.
