The Precisely as Designed. The Reply Was Nonetheless Unsuitable.
I need to inform you in regards to the second I ended trusting retrieval scores.
I used to be operating a question in opposition to a data base I had constructed fastidiously. Good chunking. Hybrid search. Reranking. The highest-k paperwork got here again with cosine similarities as excessive as 0.86. Each indicator mentioned the pipeline was working. I handed these paperwork to a QA mannequin, received a assured reply, and moved on.
The reply was flawed.
Not hallucinated-wrong. Not retrieval-failed-wrong. The precise paperwork had come again. Each of them. A preliminary earnings determine and the audited revision that outmoded it, sitting aspect by aspect in the identical context window. The mannequin learn each, selected one, and reported it with 80% confidence. It had no mechanism to inform me it had been requested to referee a dispute it was by no means designed to guage.
That’s the failure mode this text is about. It doesn’t present up in your retrieval metrics. It doesn’t set off your hallucination detectors. It lives within the hole between context meeting and technology — the one step within the RAG pipeline that just about no person evaluates.
I constructed a reproducible experiment to isolate it. Every part on this article runs on a CPU in about 220 MB. No API key. No cloud. No GPU. The output you see within the terminal screenshots is unmodified.
Full Supply Code: https://github.com/Emmimal/rag-conflict-demo
What the Experiment Exams
The setup is intentionally scientific. Three questions. One data base containing three conflicting doc pairs that make immediately contradictory claims about the identical reality. Retrieval is tuned to return each conflicting paperwork each time.
The query is just not whether or not retrieval works. It does. The query is: what does the mannequin do once you hand it a contradictory transient and ask it to reply with confidence?
The reply, as you will note, is that it picks a aspect. Silently. Confidently. With out telling you it had a option to make.
Three Situations, Every Drawn from Manufacturing
State of affairs A — The restatement no person informed the mannequin about
An organization’s This autumn earnings launch reviews annual income of $4.2M for fiscal yr 2023. Three months later, exterior auditors restate that determine to $6.8M. Each paperwork dwell within the data base. Each are listed. When somebody asks “What was Acme Corp’s income for fiscal yr 2023?” — each come again, with similarity scores of 0.863 and 0.820 respectively.
The mannequin solutions $4.2M.
It selected the preliminary determine over the audited revision as a result of the preliminary doc scored marginally greater in retrieval. Nothing in regards to the reply indicators {that a} extra authoritative supply disagreed.
State of affairs B — The coverage replace that arrived too late
A June 2023 HR coverage mandates three days per week in-office. A November 2023 revision explicitly reverses it — totally distant is now permitted. Each paperwork are retrieved (similarity scores 0.806 and 0.776) when an worker asks in regards to the present distant work coverage.
The mannequin solutions with the June coverage. The stricter, older rule. The one which now not applies.
State of affairs C — The API docs that by no means received deprecated
Model 1.2 of an API reference states a charge restrict of 100 requests per minute. Model 2.0, printed after an infrastructure improve, raises it to 500. Each are retrieved (scores 0.788 and 0.732).
The mannequin solutions 100. A developer utilizing this reply to configure their charge limiter will throttle themselves to one-fifth of their precise allowance.
None of those are edge circumstances. Each manufacturing data base accumulates precisely these patterns over time: monetary restatements, coverage revisions, versioned documentation. The pipeline has no layer that detects or handles them.
Working the Experiment
pip set up -r necessities.txt
python rag_conflict_demo.pynecessities.txt
sentence-transformers>=2.7.0 # all-MiniLM-L6-v2 (~90 MB)
transformers>=4.40.0 # deepset/minilm-uncased-squad2 (~130 MB)
torch>=2.0.0 # CPU-only is okay
numpy>=1.24.0
colorama>=0.4.6Two fashions. One for embeddings, one for extractive QA. Each obtain robotically on first run and cache domestically. Complete: ~220 MB. No authentication required.
Part 1: What Naive RAG Does
Right here is the unmodified terminal output from Part 1 — normal RAG with no battle dealing with:
────────────────────────────────────────────────────────────────────
NAIVE | State of affairs A — Numerical Battle
────────────────────────────────────────────────────────────────────
Question : What was Acme Corp's annual income for fiscal yr 2023?
Reply : $4.2M
Confidence : 80.3%
Battle : YES — see warning
Sources retrieved
[0.863] This autumn-2023-Earnings-Launch (2024-01-15)
[0.820] 2023-Annual-Report-Revised (2024-04-03)
[0.589] Firm-Overview-2024 (2024-01-01)
Battle pairs
fin-001 ↔ fin-002
numerical contradiction (topic_sim=0.83)
[Q4-2023-Earnings-Release: {'$4.2M'}] vs [2023-Annual-Report-Revised: {'$6.8M'}]
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
NAIVE | State of affairs B — Coverage Battle
────────────────────────────────────────────────────────────────────
Question : What's the present distant work coverage for workers?
Reply : all staff are required to be current within the workplace
a minimal of three days per week
Confidence : 78.3%
Battle : YES — see warning
Sources retrieved
[0.806] HR-Coverage-June-2023 (2023-06-01)
[0.776] HR-Coverage-November-2023 (2023-11-15)
[0.196] HR-Coverage-November-2023 (2023-11-15)
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
NAIVE | State of affairs C — Technical Battle
────────────────────────────────────────────────────────────────────
Question : What's the API charge restrict for the usual tier?
Reply : 100 requests per minute
Confidence : 81.0%
Battle : YES — see warning
Sources retrieved
[0.788] API-Reference-v1.2 (2023-02-10)
[0.732] API-Reference-v2.0 (2023-09-20)
[0.383] API-Reference-v2.0 (2023-09-20)
────────────────────────────────────────────────────────────────────
Three questions. Three flawed solutions. Confidence between 78% and 81% on each one among them.
Discover what is going on within the logs earlier than every response:
09:02:20 | WARNING | Battle detected: {('fin-001', 'fin-002'): "numerical contradiction..."}
09:02:24 | WARNING | Battle detected: {('hr-001', 'hr-002'): "contradiction sign asymmetry..."}
09:02:25 | WARNING | Battle detected: {('api-001', 'api-002'): "contradiction sign asymmetry..."}The conflicts are detected. They’re logged. After which, as a result of resolve_conflicts=False, the pipeline passes the total contradictory context to the mannequin and solutions anyway. That warning goes nowhere. In a manufacturing system with no battle detection layer, you wouldn’t even get the warning.
Why the Mannequin Behaves This Method
This requires a second of rationalization, as a result of the mannequin is just not damaged. It’s doing precisely what it was skilled to do.
deepset/minilm-uncased-squad2 is an extractive QA mannequin. It reads a context string and selects the span with the best mixed start-logit and end-logit rating. It has no output class for “I see two contradictory claims.” When the context incorporates each $4.2M and $6.8M, the mannequin computes token-level scores throughout your complete string and selects whichever span wins.
That choice is pushed by components that don’t have anything to do with correctness [8]. The 2 major drivers are:
Place bias. Earlier spans within the context obtain marginally greater consideration scores because of the encoder structure. The preliminary doc ranked greater in retrieval and due to this fact appeared first.
Language energy. Direct declarative statements (“income of $4.2M”) outscore hedged or conditional phrasing (“following restatement… is $6.8M”).
A 3rd contributing issue is lexical alignment — spans whose vocabulary overlaps extra intently with the query tokens rating greater no matter whether or not the underlying declare is present or authoritative.
Critically, what the mannequin does not take into account in any respect: supply date, doc authority, audit standing, or whether or not one declare supersedes one other. These indicators are merely invisible to the extractive mannequin.
The identical dynamic performs out in generative LLMs, however much less visibly — the mannequin paraphrases relatively than extracting verbatim spans, so the flawed reply is wearing fluent prose. The mechanism is identical. Joren et al. (2025) display at ICLR 2025 that frontier fashions together with Gemini 1.5 Professional, GPT-4o, and Claude 3.5 regularly produce incorrect solutions relatively than abstaining when retrieved context is inadequate to reply the question — and that this failure is just not mirrored within the mannequin’s expressed confidence.
The failure is just not a mannequin deficiency. It’s an architectural hole: the pipeline has no stage that detects contradictions earlier than handing context to technology.
Constructing the Battle Detection Layer
The detector sits between retrieval and technology. It examines each pair of retrieved paperwork and flags contradictions earlier than the QA mannequin sees the context. Crucially, embeddings for all retrieved paperwork are computed in a single batched ahead cross earlier than pair comparability begins — every doc is encoded precisely as soon as, no matter what number of pairs it participates in.
Two heuristics do the work.
Heuristic 1: Numerical Contradiction
Two topic-similar paperwork that include non-overlapping significant numbers are flagged. The implementation filters out years (1900–2099) and naked small integers (1–9), which seem ubiquitously in enterprise textual content and would generate fixed false positives if handled as declare values.
@classmethod
def _extract_meaningful_numbers(cls, textual content: str) -> set[str]:
outcomes = set()
for m in cls._NUM_RE.finditer(textual content):
uncooked = m.group().strip()
numeric_core = re.sub(r"[$€£MBK%,]", "", uncooked, flags=re.IGNORECASE).strip()
attempt:
val = float(numeric_core)
besides ValueError:
proceed
if 1900 <= val <= 2099 and "." not in numeric_core:
proceed # skip years
if val < 10 and re.fullmatch(r"d+", uncooked):
proceed # skip naked small integers
outcomes.add(uncooked)
return outcomesUtilized to State of affairs A: fin-001 yields {'$4.2M'}, fin-002 yields {'$6.8M'}. Empty intersection — battle detected.
Heuristic 2: Contradiction Sign Asymmetry
Two paperwork discussing the identical matter, the place one incorporates contradiction tokens the opposite doesn’t, are flagged. The token set splits into two teams saved as separate frozenset objects:
_NEGATION_TOKENS: “not”, “by no means”, “no”, “can not”, “doesn’t”, “isn’t”, and associated varieties_DIRECTIONAL_TOKENS: “elevated”, “decreased”, “lowered”, “eradicated”, “eliminated”, “discontinued”
These are unioned into CONTRADICTION_SIGNALS. Conserving them separate makes domain-specific tuning simple — a authorized corpus may want a broader negation set; a changelog corpus may want extra directional tokens.
Utilized to State of affairs B: hr-002 incorporates “no” (from “now not required”); hr-001 doesn’t. Asymmetry detected. Utilized to State of affairs C: api-002 incorporates “elevated”; api-001 doesn’t. Asymmetry detected.
Each heuristics require topic_sim >= 0.68 earlier than firing. This threshold gates out unrelated paperwork that occur to share a quantity or a negation phrase. The 0.68 worth was calibrated for this doc set with all-MiniLM-L6-v2 — deal with it as a place to begin, not a common fixed. Totally different embedding fashions and completely different domains would require recalibration.
The Decision Technique: Cluster-Conscious Recency
When conflicts are detected, the pipeline resolves them by holding probably the most just lately timestamped doc from every battle cluster. The important thing design choice is cluster-aware.
A top-k end result might include a number of unbiased battle clusters — two monetary paperwork disagreeing on income and two API paperwork disagreeing on charge limits, all in the identical top-3 end result. A naive strategy — maintain solely the only most up-to-date doc from the mixed conflicting set — would silently discard the profitable doc from each cluster besides probably the most just lately printed one general.
As a substitute, the implementation builds a battle graph, finds related parts through iterative DFS, and resolves every element independently:
@staticmethod
def _resolve_by_recency(
contexts: record[RetrievedContext],
battle: ConflictReport,
) -> record[RetrievedContext]:
# Construct adjacency record
adj: dict[str, set[str]] = defaultdict(set)
for a_id, b_id in battle.conflict_pairs:
adj[a_id].add(b_id)
adj[b_id].add(a_id)
# Related parts through iterative DFS
visited: set[str] = set()
clusters: record[set[str]] = []
for begin in adj:
if begin not in visited:
cluster: set[str] = set()
stack = [start]
whereas stack:
node = stack.pop()
if node not in visited:
visited.add(node)
cluster.add(node)
stack.lengthen(adj[node] - visited)
clusters.append(cluster)
all_conflicting_ids = set().union(*clusters) if clusters else set()
non_conflicting = [c for c in contexts if c.document.doc_id not in all_conflicting_ids]
resolved_docs = []
for cluster in clusters:
cluster_ctxs = [c for c in contexts if c.document.doc_id in cluster]
# ISO-8601 timestamps type lexicographically — max() provides most up-to-date
greatest = max(cluster_ctxs, key=lambda c: c.doc.timestamp)
resolved_docs.append(greatest)
return non_conflicting + resolved_docsNon-conflicting paperwork cross via unchanged. Every battle cluster contributes precisely one winner.
Part 2: What Battle-Conscious RAG Does
────────────────────────────────────────────────────────────────────
RESOLVED | State of affairs A — Numerical Battle
────────────────────────────────────────────────────────────────────
Question : What was Acme Corp's annual income for fiscal yr 2023?
Reply : $6.8M
Confidence : 79.6%
Battle : RESOLVED
⚠ Conflicting sources detected — reply derived from most up-to-date
doc per battle cluster.
Sources retrieved
[0.820] 2023-Annual-Report-Revised (2024-04-03)
[0.589] Firm-Overview-2024 (2024-01-01)
Battle cluster resolved: saved '2023-Annual-Report-Revised' (2024-04-03),
discarded 1 older doc(s).
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
RESOLVED | State of affairs B — Coverage Battle
────────────────────────────────────────────────────────────────────
Reply : staff are now not required to keep up
a set in-office schedule
Confidence : 78.0%
Battle : RESOLVED
Battle cluster resolved: saved 'HR-Coverage-November-2023' (2023-11-15),
discarded 1 older doc(s).
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
RESOLVED | State of affairs C — Technical Battle
────────────────────────────────────────────────────────────────────
Reply : 500 requests per minute
Confidence : 80.9%
Battle : RESOLVED
Battle cluster resolved: saved 'API-Reference-v2.0' (2023-09-20),
discarded 1 older doc(s).
────────────────────────────────────────────────────────────────────
Three questions. Three appropriate solutions. The arrogance scores are virtually similar to Part 1 — 78–81% — which underscores the unique level: confidence was by no means the sign that one thing had gone flawed. It nonetheless is just not. The one factor that modified is the structure.
What the Heuristics Can not Catch
I need to be exact in regards to the failure envelope, as a result of a way that understates its personal limitations is just not helpful.
Paraphrased conflicts. The heuristics catch numerical variations and express contradiction tokens. They won’t catch “the service was retired” versus “the service is at present out there.” That may be a actual battle with no numeric distinction and no negation token. For these, a Pure Language Inference mannequin — cross-encoder/nli-deberta-v3-small at ~80 MB — can rating entailment versus contradiction between sentence pairs. That is the extra strong path described within the tutorial literature (Asai et al., 2023), and the ConflictDetector class is designed to be prolonged on the _pair_conflict_reason methodology for precisely this objective.
Non-temporal conflicts. Recency-based decision is suitable for versioned paperwork and coverage updates. It isn’t applicable for knowledgeable opinion disagreements (the minority view could also be appropriate), cross-methodology knowledge conflicts (recency is irrelevant), or multi-perspective queries (the place surfacing each views is the suitable response). In these circumstances, the ConflictReport knowledge construction gives the uncooked materials to construct a distinct response — surfacing each claims, flagging for human assessment, or asking the consumer for clarification.
Scale. Pair comparability is O(k²) in retrieved paperwork. For ok=3 that is trivial; for ok=20 it’s nonetheless high quality. For pipelines retrieving ok=100 or extra, pre-indexing recognized battle pairs or cluster-based detection turns into vital.
The place the Analysis Neighborhood Is Taking This
What you’ve gotten seen here’s a sensible heuristic approximation of an issue that lively analysis is attacking at a way more refined degree.
Cattan et al. (2025) launched the CONFLICTS benchmark — the primary particularly designed to trace how fashions deal with data conflicts in sensible RAG settings. Their taxonomy identifies 4 battle classes — freshness, conflicting opinions, complementary data, and misinformation — every requiring distinct mannequin behaviour. Their experiments present that LLMs regularly fail to resolve conflicts appropriately throughout all classes, and that explicitly prompting fashions to cause about potential conflicts considerably improves response high quality, although substantial room for enchancment stays.
Ye et al. (2026) launched TCR (Clear Battle Decision), a plug-and-play framework that disentangles semantic relevance from factual consistency through twin contrastive encoders. Self-answerability estimation gauges confidence within the mannequin’s parametric reminiscence, and the ensuing scalar indicators are injected into the generator through light-weight soft-prompt tuning. Throughout seven benchmarks, TCR improves battle detection by 5–18 F1 factors whereas including solely 0.3% parameters.
Gao et al. (2025) launched CLEAR (Battle-Localized and Enhanced Consideration for RAG), which probes LLM hidden states on the sentence illustration degree to detect the place conflicting data manifests internally. Their evaluation reveals that data integration happens hierarchically and that conflicting versus aligned data reveals distinct distributional patterns inside sentence-level representations. CLEAR makes use of these indicators for conflict-aware fine-tuning that guides the mannequin towards correct proof integration.
The constant discovering throughout all of this work matches what this experiment demonstrates immediately: retrieval high quality and reply high quality are distinct dimensions, and the hole between them is bigger than the group has traditionally acknowledged.
The distinction between that analysis and this text is 220 MB and no authentication.
What You Ought to Truly Do With This
1. Add a battle detection layer earlier than technology. The ConflictDetector class is designed to drop into an present pipeline on the level the place you assemble your context string. Even the 2 easy heuristics right here will catch the patterns that seem most frequently in enterprise corpora: restatements, coverage updates, versioned documentation.
2. Distinguish battle sorts earlier than resolving. A temporal battle (use the newer doc) is a distinct downside from a factual dispute (flag for human assessment) or an opinion battle (floor each views). A single decision technique utilized blindly creates new failure modes.
3. Log each ConflictReport. After every week of manufacturing site visitors you’ll understand how usually your particular corpus generates conflicting retrieved units, which doc pairs battle most regularly, and what question patterns set off conflicts. That knowledge is extra actionable than any artificial benchmark.
4. Floor uncertainty once you can not resolve it. The precise reply to an unresolvable battle is to not decide one and conceal the selection. The warning subject in RAGResponse is there exactly to help responses like: “I discovered conflicting data on this matter. The June 2023 coverage states X; the November 2023 replace states Y. The November doc is newer.”
Working the Full Demo
# Full output with INFO logs
python rag_conflict_demo.py
# Demo output solely (suppress mannequin loading logs)
python rag_conflict_demo.py --quiet
# Run unit assessments with out downloading fashions
python rag_conflict_demo.py --test
# Plain terminal output for log seize / CI
python rag_conflict_demo.py --no-colorAll output proven on this article is unmodified output from an area Home windows machine operating Python 3.9+ in a digital atmosphere. The code and output are totally reproducible by any reader with the listed dependencies put in.
The Takeaway
The retrieval downside is essentially solved. Vector search is quick, correct, and well-understood. The group has spent years optimising it.
The context-assembly downside is just not solved. No one is measuring it. The hole between “appropriate paperwork retrieved” and “appropriate reply produced” is actual, it’s common, and it produces assured flawed solutions with no sign that something went flawed.
The repair doesn’t require a bigger mannequin, a brand new structure, or extra coaching. It requires one extra pipeline stage, operating on embeddings you have already got, at zero marginal latency.
The experiment above runs in about thirty seconds on a laptop computer. The query is whether or not your manufacturing system has the equal layer — and if not, what it’s silently answering flawed proper now.
References
[1] Ye, H., Chen, S., Zhong, Z., Xiao, C., Zhang, H., Wu, Y., & Shen, F. (2026). Seeing via the battle: Clear data battle dealing with in retrieval-augmented technology. arXiv:2601.06842. https://doi.org/10.48550/arXiv.2601.06842
[2] Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023). Self-RAG: Studying to retrieve, generate, and critique via self-reflection. arXiv:2310.11511. https://doi.org/10.48550/arXiv.2310.11511
[3] Cattan, A., Jacovi, A., Ram, O., Herzig, J., Aharoni, R., Goldshtein, S., Ofek, E., Szpektor, I., & Caciularu, A. (2025). DRAGged into conflicts: Detecting and addressing conflicting sources in search-augmented LLMs. arXiv:2506.08500. https://doi.org/10.48550/arXiv.2506.08500
[4] Gao, L., Bi, B., Yuan, Z., Wang, L., Chen, Z., Wei, Z., Liu, S., Zhang, Q., & Su, J. (2025). Probing latent data battle for devoted retrieval-augmented technology. arXiv:2510.12460. https://doi.org/10.48550/arXiv.2510.12460
[5] Jin, Z., Cao, P., Chen, Y., Liu, Okay., Jiang, X., Xu, J., Li, Q., & Zhao, J. (2024). Tug-of-war between data: Exploring and resolving data conflicts in retrieval-augmented language fashions. arXiv:2402.14409. https://doi.org/10.48550/arXiv.2402.14409
[6] Joren, H., Zhang, J., Ferng, C.-S., Juan, D.-C., Taly, A., & Rashtchian, C. (2025). Ample context: A brand new lens on retrieval augmented technology techniques. arXiv:2411.06037. https://doi.org/10.48550/arXiv.2411.06037
[7] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., … & Kiela, D. (2020). Retrieval-augmented technology for knowledge-intensive NLP duties. arXiv:2005.11401. https://doi.org/10.48550/arXiv.2005.11401
[8] Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., & Hajishirzi, H. (2023). When to not belief language fashions: Investigating effectiveness of parametric and non-parametric reminiscences. arXiv:2212.10511. https://doi.org/10.48550/arXiv.2212.10511
[9] Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings utilizing Siamese BERT-networks. arXiv:1908.10084. https://doi.org/10.48550/arXiv.1908.10084
[10] Xu, R., Qi, Z., Guo, Z., Wang, C., Wang, H., Zhang, Y., & Xu, W. (2024). Data conflicts for LLMs: A survey. arXiv:2403.08319. https://doi.org/10.48550/arXiv.2403.08319
[11] Xie, J., Zhang, Okay., Chen, J., Lou, R., & Su, Y. (2023). Adaptive chameleon or cussed sloth: Revealing the habits of huge language fashions in data conflicts. arXiv:2305.13300. https://doi.org/10.48550/arXiv.2305.13300
Full Supply Code: https://github.com/Emmimal/rag-conflict-demo
Fashions Used
Each fashions obtain robotically on first run and cache domestically. No API key or HuggingFace authentication is required.
Disclosure
All code was written, debugged, and validated by the writer via a number of iterations of actual execution. All terminal output on this article is unmodified output from an area Home windows machine operating Python 3.9+ in a digital atmosphere. The code and output are totally reproducible by any reader with the listed dependencies put in.
The writer has no monetary relationship with Hugging Face, deepset, or any organisation referenced on this article. Mannequin and library selections had been made solely on the premise of measurement, licence, and CPU compatibility.
