Every new model generation arrives with fanfare: better benchmarks, higher accuracy scores, more impressive demos. GPT-5 reasons through complex problems. Claude plans ahead when writing poetry. Gemini processes images and video with startling fluency. The intelligence curve keeps climbing.<\/p>\n
But there’s a second curve that rarely makes the keynote slides \u2014 the honesty curve. And it’s barely moved.<\/p>\n
This isn’t a vague philosophical complaint. It’s a structural problem baked into how these models are trained, evaluated, and deployed. And it’s one that hits hardest in exactly the kind of work where people increasingly rely on AI: extracting data from contracts, parsing invoices, summarizing meeting notes, building CRM records from messy inputs.<\/p>\n
This post unpacks why the intelligence-honesty gap exists, what the latest research tells us about its causes, and \u2014 most practically \u2014 three prompt rules you can apply today to force AI to be honest about what it doesn’t know.<\/p>\n
When we say a model “got smarter,” we usually mean it scores higher on benchmarks \u2014 math competitions, coding challenges, multi-step reasoning tasks. These are real improvements. But benchmark scores measure a model’s ability to produce correct answers. They don’t measure a model’s willingness to say “I don’t know.”<\/p>\n
In fact, the incentive structure actively punishes honesty.<\/p>\n
In September 2025, OpenAI published a\u00a0research paper<\/a>\u00a0that made this problem precise. The team \u2014 including researchers from Georgia Tech \u2014 examined major AI benchmarks and found that the vast majority use binary grading: either the answer is correct and gets a point, or it’s wrong and gets zero. Crucially, abstaining \u2014 saying “I don’t know” \u2014 also gets zero. The mathematical consequence is straightforward: guessing always has a higher expected score than abstaining. A model that bluffs on every uncertain question will rank higher than one that honestly declines.<\/p>\n OpenAI’s own blog post put it plainly: the situation is like a multiple-choice test where leaving an answer blank guarantees a zero, but guessing at least gives you a chance. Under those rules, the rational strategy is to always guess \u2014 even when you have no idea. And that’s exactly what the models learn to do.<\/p>\n The paper demonstrated this with a striking example: when asked for the PhD dissertation title of one of its own co-authors, a widely-used model confidently produced three different titles across three attempts. All three were wrong. It did the same with his birthday \u2014 three dates, all incorrect, all delivered with unwavering confidence.<\/p>\n This isn’t a bug that can be patched. It’s the natural outcome of optimizing for accuracy-only metrics. As the OpenAI researchers argue, the mainstream benchmarks and leaderboards need to be redesigned to penalize confident errors more heavily than uncertainty. Until that happens, every model that climbs the leaderboard does so in part by learning to bluff better.<\/p>\n The OpenAI paper explains the\u00a0incentive<\/em>\u00a0problem. But what happens mechanically inside the model when it makes something up?<\/p>\n Anthropic’s interpretability research \u2014 published in March 2025 under the title “Tracing the Thoughts of a Large Language Model<\/a>” \u2014 provides some of the most detailed answers we have. Using what they describe as a “microscope” for AI, Anthropic’s team traced the internal circuits that activate when Claude processes a question. It’s worth noting that these findings are specific to Claude 3.5 Haiku \u2014 other model families may handle uncertainty through different internal mechanisms \u2014 but the patterns are likely general enough to be instructive.<\/p>\n One of their most revealing discoveries involves what we might call a default refusal mechanism. In Claude, refusing to answer is actually the\u00a0default<\/em>\u00a0behavior: the researchers found a circuit that is “on” by default and causes the model to state it has insufficient information. But when the model recognizes a “known entity” \u2014 say, Michael Jordan the basketball player \u2014 a competing set of features fires up and\u00a0suppresses<\/em>\u00a0this default circuit, allowing the model to respond.<\/p>\n The problem arises when this mechanism misfires. If the model recognizes a name but doesn’t actually know the relevant facts, the “known entity” signal can still override the “I don’t know” circuit. The result: a confident, detailed, completely fabricated answer. In one experiment, the researchers used a person named Michael Batkin \u2014 someone unknown to the model, who by default triggered a refusal. But when they artificially activated the “known entity” features or inhibited the “can’t answer” features, Claude promptly \u2014 and consistently \u2014 hallucinated that Batkin was famous for playing chess.<\/p>\n Even more unsettling: Anthropic found evidence that when Claude can’t easily compute an answer (say, the cosine of a large number), it sometimes engages in what philosopher Harry Frankfurt would call\u00a0bullshitting<\/em>\u00a0\u2014 producing an answer without any internal evidence of the calculation actually occurring. Despite claiming to have run the math, the interpretability tools revealed no trace of any computation. When given a hint about what the answer should be, Claude worked\u00a0backwards<\/em>, constructing plausible-looking intermediate steps that lead to the hinted answer \u2014 a textbook case of motivated reasoning.<\/p>\n These findings matter because they show that the honesty problem isn’t just about training incentives. The models have internal mechanisms that are\u00a0supposed<\/em>\u00a0to catch uncertainty \u2014 but those mechanisms can be overridden by other pressures, including the drive toward grammatical coherence and the pattern-matching instinct to fill in gaps.<\/p>\n All of this would be merely academic if people treated AI output with appropriate skepticism. They don’t.<\/p>\n Automation bias \u2014 the tendency to over-rely on automated recommendations \u2014 is one of the most thoroughly documented phenomena in human-computer interaction research. A\u00a02025 systematic review<\/a>\u00a0published in\u00a0AI & Society<\/em>\u00a0analyzed 35 peer-reviewed studies spanning healthcare, finance, national security, and public administration. The pattern was consistent across domains: when an AI system delivers a confident answer, people accept it. They check less. They override their own judgment.<\/p>\n A\u00a0randomized clinical trial<\/a>\u00a0conducted with AI-trained physicians in Pakistan (published as a preprint in August 2025) made the dynamic especially clear. Even doctors who had completed 20 hours of AI-literacy training \u2014 including instruction on how to critically evaluate AI output \u2014 were vulnerable to automation bias when exposed to erroneous LLM recommendations. The training helped, but it didn’t eliminate the problem. Confident-sounding AI output has a gravitational pull that’s difficult to resist, even when you know to look for errors.<\/p>\n The real-world consequences are already visible. In February 2024, Air Canada was ordered to pay damages to a customer after a support chatbot \u2014 not a large language model, but an AI system nonetheless \u2014 hallucinated a bereavement fare policy that didn’t exist. The chatbot confidently told the customer they could retroactively request a discount within 90 days of purchase. The actual policy allowed no such thing. But the system stated it with such authority that the customer relied on it to make a financial decision. The underlying technology differed from today’s LLMs, but the dynamic was identical: confident AI output, uncritical human acceptance.<\/p>\n In an operations context, the failure modes are subtler but no less damaging. Consider a contract with payment terms mentioned on page 8 and page 14 \u2014 and the two pages say different things. A human reviewer might catch the discrepancy. An AI, asked to extract the payment terms, will pick one and move on. It won’t mention the conflict. It won’t flag the ambiguity. It will fill the cell in your spreadsheet with “Net 30” and give you no indication that page 14 says “Net 45.”<\/p>\n Meeting notes are another minefield. “Let’s circle back next week” becomes a specific date and a named owner in the AI’s summary \u2014 details that nobody actually stated, but that the model invented to produce a clean, actionable output.<\/p>\n The pattern is the same across invoices, insurance documents, lease agreements, vendor scoring, CRM data entry: wherever AI is used to extract structured information from messy sources, the model’s instinct to\u00a0fill every field<\/em>\u00a0works directly against the user’s need to know which fields are uncertain.<\/p>\n These three problems \u2014 training incentives that reward guessing, internal mechanisms that can override uncertainty detection, and human psychology that accepts confident output at face value \u2014 come from different research streams. But they converge on the same practical conclusion: by default, AI will guess rather than admit ignorance, and people will trust the guess.<\/p>\n You can’t fix the training pipeline. You can’t redesign the benchmarks. But you can change the local incentive structure inside the conversation. The following three rules \u2014 adapted from a practical\u00a0framework by D-Squared<\/a>\u00a0\u2014 do exactly that. They work because they explicitly reverse the default dynamic: instead of rewarding completeness, they reward honesty about uncertainty. Note that the effectiveness of these techniques may vary across model families \u2014 they’ve been tested primarily with ChatGPT and Claude, and other models may respond differently.<\/p>\n The single most effective change you can make is to explicitly instruct the model to leave fields blank when the data is ambiguous, missing, or unclear \u2014 and to explain why.<\/p>\n Without this rule, every field gets filled. With this rule, the model produces output like:<\/p>\n
\nWhy Models Confabulate: Insights from Interpretability Research<\/h2>\n
\nAutomation Bias: Why This Matters More Than You Think<\/h2>\n
\nThree Prompt Rules That Change the Incentive<\/h2>\n
Rule 1: Force Blank + Explain<\/h3>\n