Can AI fact-check AI? Why the blind spots are shared

It has become a common workflow: get an answer from one assistant, paste it into another, and ask, is this correct? The second model reads the fluent, confident text and replies with fluent, confident agreement. Two AIs have now signed off, and the answer feels double-checked.

What actually happened is that one system optimized to sound right reviewed the output of another system optimized to sound right. Neither opened a source. If the claim was plausible enough to be generated, it is usually plausible enough to pass, and plausibility was never the question. The ritual feels like verification because it copies the shape of one: a second opinion, a review step, a sign-off. It just skips the part where evidence gets consulted.

The Self-Correction Bench study made the comparison unusually clean. Researchers injected the exact same error into a conversation in two ways: once attributed to an external source, and once as part of the model's own earlier output. Models corrected the external version far more reliably; when the identical mistake appeared in their own words, they missed it 64.5% of the time on average across the 14 open-source, non-reasoning models tested.

A separate line of work on reasoning and planning tasks found something stronger: prompting models to critique and revise their own answers often degraded performance, with each round of self-critique compounding rather than correcting errors. And the deeper reason connects to what OpenAI's own researchers showed about why models hallucinate at all: training rewards confident answers over honest uncertainty. A checker built on the same incentive does not suddenly acquire the missing skepticism.

In court, in journalism, and in science, a second witness matters because their errors are independent. Two people who saw the event separately are unlikely to invent the same wrong detail. Independence is what makes agreement informative.

Two language models are not independent witnesses. They are trained on overlapping data, optimized by similar objectives, and rewarded by the same benchmarks for the same confident style. When a claim is the kind of thing models tend to get wrong, they tend to get it wrong together, and to affirm it in each other with equal fluency. This is also how source laundering compounds: the second model inherits the first model's confident restatement, now stripped of whatever weak sourcing produced it, and grades the polish instead of the provenance. Agreement between correlated systems is not two votes. It is one vote, counted twice.

Independence means the verdict comes from something that did not produce the claim. In practice, that is the source itself: the document, the statute, the paper, the contract. A check is independent when you open the passage the claim rests on and read whether it says what the answer says it says.

That is deliberately the cheapest step in a grounded system. When every claim carries a link to its exact passage, the independent check takes seconds: read the claim, click, compare, decide. You do not need to check everything; you need the checks you do run to be real. Scope them to the claims that drive decisions, the numbers you will repeat, and anything that surprised you, the same triage the 90-second verification ritual teaches.

None of this means a second AI is worthless. It means knowing what it can and cannot tell you. Disagreement between two models is a useful flag: if they diverge on a factual claim, at least one is wrong, and you have found exactly where to open the source. A second model is also good at stress-testing an argument, surfacing counterpoints, and spotting internal inconsistencies, because those are properties of the text itself, not of the world. What a second model cannot do is confirm. Agreement is weak evidence; the passage is strong evidence.

Tatsulok is built so the strong evidence is always the near one: answers come only from your documents and curated collections, every claim links to the exact passage behind it, and an honest gap is stated instead of papered over. The independent checker, you, gets a one-click path to the only opinion that settles it.

This week's drill, no signup needed: take one AI answer, ask a different AI to verify it, and then check one load-bearing claim yourself at the source. Score the AI verifier against the passage. Most people only need to run this once to recalibrate what agreement is worth.