Every new model generation arrives with fanfare: better benchmarks, higher accuracy scores, more impressive demos. GPT-5 reasons through complex problems. Claude plans ahead when writing poetry. Gemini processes images and video with startling fluency. The intelligence curve keeps climbing.

But there’s a second curve that rarely makes the keynote slides — the honesty curve. And it’s barely moved.

This isn’t a vague philosophical complaint. It’s a structural problem baked into how these models are trained, evaluated, and deployed. And it’s one that hits hardest in exactly the kind of work where people increasingly rely on AI: extracting data from contracts, parsing invoices, summarizing meeting notes, building CRM records from messy inputs.

This post unpacks why the intelligence-honesty gap exists, what the latest research tells us about its causes, and — most practically — three prompt rules you can apply today to force AI to be honest about what it doesn’t know.

---

The Gap: Intelligence vs. Honesty

When we say a model “got smarter,” we usually mean it scores higher on benchmarks — math competitions, coding challenges, multi-step reasoning tasks. These are real improvements. But benchmark scores measure a model’s ability to produce correct answers. They don’t measure a model’s willingness to say “I don’t know.”

In fact, the incentive structure actively punishes honesty.

In September 2025, OpenAI published a research paper that made this problem precise. The team — including researchers from Georgia Tech — examined major AI benchmarks and found that the vast majority use binary grading: either the answer is correct and gets a point, or it’s wrong and gets zero. Crucially, abstaining — saying “I don’t know” — also gets zero. The mathematical consequence is straightforward: guessing always has a higher expected score than abstaining. A model that bluffs on every uncertain question will rank higher than one that honestly declines.

OpenAI’s own blog post put it plainly: the situation is like a multiple-choice test where leaving an answer blank guarantees a zero, but guessing at least gives you a chance. Under those rules, the rational strategy is to always guess — even when you have no idea. And that’s exactly what the models learn to do.

The paper demonstrated this with a striking example: when asked for the PhD dissertation title of one of its own co-authors, a widely-used model confidently produced three different titles across three attempts. All three were wrong. It did the same with his birthday — three dates, all incorrect, all delivered with unwavering confidence.

This isn’t a bug that can be patched. It’s the natural outcome of optimizing for accuracy-only metrics. As the OpenAI researchers argue, the mainstream benchmarks and leaderboards need to be redesigned to penalize confident errors more heavily than uncertainty. Until that happens, every model that climbs the leaderboard does so in part by learning to bluff better.

---

Why Models Confabulate: Insights from Interpretability Research

The OpenAI paper explains the incentive problem. But what happens mechanically inside the model when it makes something up?

Anthropic’s interpretability research — published in March 2025 under the title “Tracing the Thoughts of a Large Language Model” — provides some of the most detailed answers we have. Using what they describe as a “microscope” for AI, Anthropic’s team traced the internal circuits that activate when Claude processes a question. It’s worth noting that these findings are specific to Claude 3.5 Haiku — other model families may handle uncertainty through different internal mechanisms — but the patterns are likely general enough to be instructive.

One of their most revealing discoveries involves what we might call a default refusal mechanism. In Claude, refusing to answer is actually the default behavior: the researchers found a circuit that is “on” by default and causes the model to state it has insufficient information. But when the model recognizes a “known entity” — say, Michael Jordan the basketball player — a competing set of features fires up and suppresses this default circuit, allowing the model to respond.

The problem arises when this mechanism misfires. If the model recognizes a name but doesn’t actually know the relevant facts, the “known entity” signal can still override the “I don’t know” circuit. The result: a confident, detailed, completely fabricated answer. In one experiment, the researchers used a person named Michael Batkin — someone unknown to the model, who by default triggered a refusal. But when they artificially activated the “known entity” features or inhibited the “can’t answer” features, Claude promptly — and consistently — hallucinated that Batkin was famous for playing chess.

Even more unsettling: Anthropic found evidence that when Claude can’t easily compute an answer (say, the cosine of a large number), it sometimes engages in what philosopher Harry Frankfurt would call bullshitting — producing an answer without any internal evidence of the calculation actually occurring. Despite claiming to have run the math, the interpretability tools revealed no trace of any computation. When given a hint about what the answer should be, Claude worked backwards, constructing plausible-looking intermediate steps that lead to the hinted answer — a textbook case of motivated reasoning.

These findings matter because they show that the honesty problem isn’t just about training incentives. The models have internal mechanisms that are supposed to catch uncertainty — but those mechanisms can be overridden by other pressures, including the drive toward grammatical coherence and the pattern-matching instinct to fill in gaps.

---

Automation Bias: Why This Matters More Than You Think

All of this would be merely academic if people treated AI output with appropriate skepticism. They don’t.

Automation bias — the tendency to over-rely on automated recommendations — is one of the most thoroughly documented phenomena in human-computer interaction research. A 2025 systematic review published in AI & Society analyzed 35 peer-reviewed studies spanning healthcare, finance, national security, and public administration. The pattern was consistent across domains: when an AI system delivers a confident answer, people accept it. They check less. They override their own judgment.

A randomized clinical trial conducted with AI-trained physicians in Pakistan (published as a preprint in August 2025) made the dynamic especially clear. Even doctors who had completed 20 hours of AI-literacy training — including instruction on how to critically evaluate AI output — were vulnerable to automation bias when exposed to erroneous LLM recommendations. The training helped, but it didn’t eliminate the problem. Confident-sounding AI output has a gravitational pull that’s difficult to resist, even when you know to look for errors.

The real-world consequences are already visible. In February 2024, Air Canada was ordered to pay damages to a customer after a support chatbot — not a large language model, but an AI system nonetheless — hallucinated a bereavement fare policy that didn’t exist. The chatbot confidently told the customer they could retroactively request a discount within 90 days of purchase. The actual policy allowed no such thing. But the system stated it with such authority that the customer relied on it to make a financial decision. The underlying technology differed from today’s LLMs, but the dynamic was identical: confident AI output, uncritical human acceptance.

In an operations context, the failure modes are subtler but no less damaging. Consider a contract with payment terms mentioned on page 8 and page 14 — and the two pages say different things. A human reviewer might catch the discrepancy. An AI, asked to extract the payment terms, will pick one and move on. It won’t mention the conflict. It won’t flag the ambiguity. It will fill the cell in your spreadsheet with “Net 30” and give you no indication that page 14 says “Net 45.”

Meeting notes are another minefield. “Let’s circle back next week” becomes a specific date and a named owner in the AI’s summary — details that nobody actually stated, but that the model invented to produce a clean, actionable output.

The pattern is the same across invoices, insurance documents, lease agreements, vendor scoring, CRM data entry: wherever AI is used to extract structured information from messy sources, the model’s instinct to fill every field works directly against the user’s need to know which fields are uncertain.

---

Three Prompt Rules That Change the Incentive

These three problems — training incentives that reward guessing, internal mechanisms that can override uncertainty detection, and human psychology that accepts confident output at face value — come from different research streams. But they converge on the same practical conclusion: by default, AI will guess rather than admit ignorance, and people will trust the guess.

You can’t fix the training pipeline. You can’t redesign the benchmarks. But you can change the local incentive structure inside the conversation. The following three rules — adapted from a practical framework by D-Squared — do exactly that. They work because they explicitly reverse the default dynamic: instead of rewarding completeness, they reward honesty about uncertainty. Note that the effectiveness of these techniques may vary across model families — they’ve been tested primarily with ChatGPT and Claude, and other models may respond differently.

Rule 1: Force Blank + Explain

The single most effective change you can make is to explicitly instruct the model to leave fields blank when the data is ambiguous, missing, or unclear — and to explain why.

Without this rule, every field gets filled. With this rule, the model produces output like:

Field	Value	Reason
Payment Terms	— BLANK	Pages 8 and 14 state different terms — net 30 vs net 45
Renewal Date	Jan 15, 2027	—
Liability Cap	— BLANK	References “Exhibit B” — not included in document

The blank fields are where the value is. They tell you exactly where to focus your attention. They’re the model admitting “I’m not sure” — something it would never do without explicit instruction.

The prompt language:

Extract the following fields from this document into a table. Rules: Only extract values that are explicitly stated in the document. When a value is ambiguous, missing, or unclear, leave the field BLANK. Add a column labeled “Reason.” Next to every blank field, include a one-sentence explanation of why you left it blank. Base every value on what the document actually says. Quote or reference the specific section you pulled it from.

One way to think about why this works is through the lens of Anthropic’s interpretability findings. The model has internal mechanisms for recognizing uncertainty — the default refusal behavior described above. But those mechanisms get overridden by the pressure to produce complete, coherent output. The “Force Blank” instruction may effectively give the uncertainty pathway permission to activate, rather than being suppressed by the completion instinct. We don’t know for certain that this is the internal mechanism at work — but the practical result is consistent and reliable.

Rule 2: Penalize Guessing

By default, from the model’s perspective, a wrong answer and a blank answer carry equal weight — neither earns praise, neither triggers correction. The model has no reason to prefer one over the other, so it defaults to guessing (which at least has a chance of being right).

Rule 2 changes this calculus with a single sentence:

A wrong answer is 3× worse than a blank. When in doubt, leave it blank.

This mirrors the scoring reform that OpenAI’s September 2025 paper advocates at the benchmark level. The researchers propose that evaluation systems should award points for correct answers, penalize wrong answers more heavily than abstentions, and give partial credit for appropriate expressions of uncertainty. They note that some standardized human exams have used this approach for decades — penalizing wrong guesses more heavily than skipped questions — precisely to discourage blind guessing.

You can’t change the benchmark. But you can embed the same incentive structure in your prompt. The 3× multiplier is arbitrary — pick any number that makes the model understand that silence is preferable to fabrication. The key insight is that you need to say it explicitly. The model won’t infer this preference on its own.

Rule 3: Show the Source

Even models that are told to “extract only” will drift toward inference. They’ll compute a renewal date from a start date and term length. They’ll estimate a total from line items. They’ll infer a contact person from an email signature. These aren’t necessarily wrong — but they’re not extraction, and the user needs to know the difference.

Rule 3 requires the model to label every value as EXTRACTED (directly stated in the document) or INFERRED (derived, calculated, or interpreted), with an explanation for every inferred value.

The prompt language:

For each field, add a column called “Source.” Mark each value as one of: EXTRACTED — directly stated in the document, exact match. INFERRED — derived from context, calculated, or interpreted. For every INFERRED field, include a one-sentence explanation of what you based it on.

The output looks like this:

Field	Value	Source	Evidence
Start Date	Jan 15, 2025	EXTRACTED	Section 2.1, paragraph 1
Term Length	24 months	EXTRACTED	Section 2.1, paragraph 2
Renewal Date	Jan 15, 2027	INFERRED	Calculated 24 months from start date. Check Section 8 — early termination clause may alter this.

The EXTRACTED/INFERRED distinction is a practical implementation of what hallucination researchers call “provenance tracking” — tying every claim back to its source. The model is perfectly capable of making this distinction; it just doesn’t bother unless you ask.

---

The Combined Prompt

All three rules work together. Here’s the complete version:

Extract the following fields from this document into a table.

Rules:

– Only extract values explicitly stated in the document.

– When a value is ambiguous, missing, or unclear, leave the field BLANK.

– A wrong answer is 3× worse than a blank. When in doubt, leave it blank.

– For each field with a value, add a “Source” column: EXTRACTED = directly stated, exact match. INFERRED = derived, calculated, or interpreted.

– For every INFERRED field, add a one-sentence explanation.

– For every BLANK field, add a row to a separate “Flags” table explaining why the value could not be extracted.

The workflow change this enables is significant. Instead of reviewing every extracted value (which nobody actually does), you review only the blanks and the inferred fields. Everything marked EXTRACTED with a section reference can be trusted at a higher confidence level. Your attention goes where it matters.

---

The Bigger Picture

These three rules are a stopgap. They work — sometimes remarkably well — but they’re fighting against the grain of how models are trained. The deeper fix requires changes at the infrastructure level.

OpenAI’s hallucination paper calls for benchmark reform: scoring systems that reward calibrated uncertainty instead of confident guessing. Anthropic’s interpretability work points toward architectural insights — understanding the internal circuits well enough to strengthen the “I don’t know” pathway rather than relying on prompt-level patches.

Perhaps the most structurally promising direction is OpenAI’s “Confessions” research (2025). Instead of relying on users to prompt honesty, the Confessions approach separates the honesty objective from the performance objective during training itself. After producing a main answer — optimized for all the usual factors like correctness, style, and helpfulness — the model generates a separate “confession” report. This report is scored exclusively on honesty: Did the model flag its uncertainties? Did it acknowledge where it took shortcuts? Crucially, nothing in the confession is held against the main answer’s score, so the model has no incentive to hide its doubts. If this approach scales, it could move the honesty problem from something users have to prompt-engineer around to something the model handles natively.

These are promising directions, but none of them are available to you today. What is available is the ability to change the local incentive structure in your prompts. Force blanks. Penalize guessing. Require source labels. These three rules won’t make AI honest by nature, but they create an environment where honesty is the path of least resistance — and that turns out to be surprisingly effective.

The models are smart enough to know when they’re guessing. They just need permission to say so.

---

Sources and Further Reading

• OpenAI (September 2025): “Why Language Models Hallucinate.” Research paper arguing that standard training and evaluation procedures reward guessing over acknowledging uncertainty.
• OpenAI (2025): “How Confessions Can Keep Language Models Honest.” Research on training models to produce separate honesty reports, scored independently from main responses.
• Anthropic (March 2025): “Tracing the Thoughts of a Large Language Model.” Interpretability research revealing internal circuits for refusal, known-entity recognition, and hallucination in Claude 3.5 Haiku.
• Anthropic (March 2025): “On the Biology of a Large Language Model.” Companion paper on circuit tracing and attribution graphs.
• Carnat, I. (November 2024): “Human, All Too Human: Accounting for Automation Bias in Generative Large Language Models.” International Data Privacy Law, Vol. 14, Issue 4, pp. 299–314.
• Qazi, I.A. et al. (August 2025): “Automation Bias in LLM Assisted Diagnostic Reasoning Among AI-Trained Physicians.” Randomized clinical trial, medRxiv preprint.
• AI & Society (July 2025): “Exploring Automation Bias in Human–AI Collaboration.” Systematic review of 35 studies.
• D-Squared (2025): “ChatGPT and Claude Got Smarter. Not More Honest.” Original slide deck presenting the three prompt rules.