The guide · Part III · Chapter 12 of 14
Chapter 12 - When AI Sounds Right but Means Wrong
From Clinical Meaning in the Age of Health Data & AI - a guide by Dr Balsam Khalil, MD. First edition, 2026.
Part III of four, chapter three of three. Chapters 10 and 11 established what a model inherits and who now writes the record. This chapter takes up the failure that is hardest to see: output that reads as correct and means something else.
Why is a plausible clinical output not a safe one - and what does fluency do to the judgement of the physician reading it?
The Clinical Problem
A patient with a long and complicated history is admitted under a team that has never met her. Her record runs to hundreds of pages, across two decades and four specialties. The team uses a summarisation tool to get oriented.
The summary is good - genuinely good. Chronological, well organised, measured in tone, appropriately hedged where the record itself is ambiguous. Fourteen items. Ninety seconds to read, and it would have taken an hour to assemble by hand.
One of the fourteen says she has a documented history of heparin-induced thrombocytopenia.
What the record actually contains, spread across three admissions in 2021, is this: heparin-induced thrombocytopenia was suspected, the antibody test came back negative, it was formally excluded, and the thrombocytopenia was attributed to sepsis.
The summary is right about thirteen things. The fourteenth changes her anticoagulation.
Nobody goes back to the source. Not out of negligence - out of the entirely rational behaviour of people who used a summary precisely so that they would not have to read twenty years of notes. That was the purpose. The summary succeeded at its purpose.
There is no moment in this scenario where a competent clinician does something wrong, and no signal for anyone to act on. The summary does not feel uncertain, because it is not uncertain. It is confident, and thirteen-fourteenths right, and beautifully written. Something has failed. It is not the model, and it is not the clinicians.
What you should see differently - by the end of this chapter
- You should understand why fluency was a valid proxy for clinical reasoning for the entire history of medicine, and why it no longer is.
- You should be able to explain why the modifiers are exactly where fluency fails, and why that is structural rather than bad luck.
- You should know what the evidence shows about how clinicians behave when given incorrect machine advice, including the parts that are unresolved.
- You should understand why improving an output's accuracy can make its verification problem worse.
- You should have one concrete method for reading an AI-generated clinical output, and understand why it targets what it targets.
Chapter 12 · Scientific foundation
The Broken Proxy and the Verification Paradox
A Proxy That Worked for Two Thousand Years
Physicians read for coherence. We always have. A well-constructed case presentation is evidence of good reasoning. A referral letter that moves logically from history to examination to assessment tells you something reliable about the person who wrote it. A junior whose summary is muddled is usually a junior whose thinking is muddled, and we correct the thinking rather than the prose.
That is not a bias. It is a sound inference, and it was sound for one specific reason: producing coherent clinical language and reasoning clinically were the same activity. There was no way to generate a fluent, well-organised, appropriately hedged clinical account without having reasoned your way to it. Fluency sat downstream of judgement, which is what made it evidence of judgement.
Generative systems break that link. Fluency is now producible without the reasoning that used to be required to produce it. So the proxy is broken. And physicians have no replacement for it, because we were never taught one - nobody needed to teach us, since the correlation held for as long as medicine has existed.
Why the Modifiers Are Exactly Where This Fails
Six properties surround every clinical concept: polarity, certainty, temporality, subject of record, laterality, episodicity. Part II followed them through every layer of the ecosystem and found them the most fragile thing in it. Now notice how they are expressed in language. They ride on small words:
- No
- Possible
- In 2019
- Her mother's
- Left
- Excluded
The concept in a clinical statement occupies the conspicuous part of the sentence - the diagnosis, the finding, the drug. The modifiers occupy the least conspicuous part: a negative particle, a hedge, a prepositional phrase, a possessive.
A generative system reproduces those small words with complete grammatical competence. What grammatical competence does not guarantee is that the small word attached to the right concept, or survived a chain of summarisation intact, or reflects what the source record actually said.
"Fluency is a property of the surface. The modifiers live on the surface too. The one signal a reader has for detecting error is generated by the same process that carries the thing most likely to be wrong."
That is why the failure in the opening scenario is invisible. Changing excluded to documented history of is three words in a summary of several hundred. It costs nothing in fluency. It inverts the clinical meaning.
What Clinicians Actually Do with Wrong Machine Advice
The evidence here is better developed than for anything else in Part III, and it deserves stating carefully - including the places where it disagrees with itself.
Expertise reduces the effect without removing it
In a controlled study where radiologists read mammograms alongside suggestions from a purported AI system, inexperienced, moderately experienced and very experienced readers were all measurably influenced by incorrect suggestions. The least experienced were significantly more affected than the very experienced (mean bias 4.0 ± 1.8 vs 1.2 ± 0.8, P = .009). But no group was unaffected. Seniority is protective; it is not protection.
The effect is substantial
In a simulation reader study where AI recommendations were deliberately altered by one category in 30% of examinations, automation bias occurred in 36.1% of the altered cases and anchoring bias in 33.9%. When the system was wrong, radiologists followed it roughly a third of the time. The gradient runs the wrong way for safety: the clinicians most affected are those with the least experience - who in most health systems are also the ones working with the least supervision, at the least sociable hours.
The benefit is real
On the examinations where the AI output had not been altered, diagnostic accuracy improved from 86.2% unaided to 90.1% with the system and explanations available. The problem is not that these systems make clinicians worse on average. It is that the average conceals a class of cases where they make them confidently wrong. There is a structural problem that accuracy alone cannot solve.
The Verification Paradox
This is the hardest idea in the chapter, and it is structural rather than technical.
The value of a summary, a draft, or a recommendation comes from saving the reader the work of consulting the source. That saving is the product. Verification is exactly that work. So an output that must be checked against the source has no value, and an output that is not checked has no safeguard. Those are not two options to choose between. They are the same axis, with utility at one end and verification at the other.
And improving accuracy makes it worse rather than better. A system that is right 70% of the time gets checked. A system that is right 98% of the time does not - and the reader is correct not to check it, most of the time. Their confidence is justified by experience.
This brings us to a property that has no name in most evaluation frameworks - but should.
A Note on Retrieval-Augmented Generation
A reasonable objection to the verification paradox runs as follows: newer systems do not generate from memory alone. They retrieve - they locate the source passage, attach a citation, and show the reader where the claim came from. Does this not restore the link between fluency and verifiability?
It does not. And the reason is the same reason that runs through every chapter of this guide. Retrieval-augmented generation resolves the question of whether the system found a source. It does not resolve the question of whether the source answers the question being asked of it. A system that retrieves a classification code and presents it as a clinical fact about this patient has found a real source. The source is correct for its purpose. The purpose is not the one the reader assumes.
The verification paradox is not about whether a citation exists. It is about whether the reader can assess, without consulting the source in full, whether the source's purpose matches the use being made of it. That assessment requires exactly the knowledge this guide has been building since Chapter 4. A link does not supply it.
Cognitive Safety - The Property Nobody Measures
Cognitive safety - the property nobody measures
A term is emerging in the literature for what this chapter describes: cognitive safety - the effect an AI system has on the reasoning of the clinician who reads it. This is a distinct property from accuracy, and none of the metrics used to evaluate these systems measures it.
Sensitivity, specificity, area under the curve, reading time - none of them says anything about whether an output makes an error more persuasive, anchors a reader before she has formed her own view, or suppresses the discomfort that would otherwise have triggered a second opinion.
A confident system can improve a diagnosis. A confident system can also make an error more convincing. The same property produces both, and only one of them is currently measured.
Chapter 12 · Clinical perspective & questions
The Clinical Perspective & The Physician's Apparatus
We Are Unusually Vulnerable - Because Our Training Worked
Medicine spends years teaching us to read for coherence and to treat a well-formed account as evidence of sound reasoning. That training was correct. It made us better clinicians. It also installed precisely the heuristic that generative systems now defeat. We are not vulnerable because we are credulous - we are vulnerable because we learned something true that has recently stopped being true.
Think about how a clinician actually notices a problem. Something does not fit. The story is inconsistent. The examination does not match the history. We say I'm not happy with this, and then we ask a colleague, or re-examine, or wait and review. Every one of those is triggered by a feeling of wrongness.
A fluent, confident, well-organised, mostly-correct output produces no such feeling. It produces the opposite of it. The failure mode described in this chapter bypasses the only detection system we have, and it does so without anyone designing it to.
The most portable thing in this guide
The Five Questions
When reading any AI-generated clinical output, do not ask is this good? Ask five questions:
- Is anything negated?
- Is anything hedged?
- Whose is it?
- When was it?
- Which side?
They take seconds. They target exactly where fluency fails and exactly where your own judgement is least engaged. And they are questions you are already equipped to answer, because they are the questions you would have been asking anyway if you had written the note yourself. It is the most portable thing in this guide, and it needs no technical knowledge, no governance authority, and nobody's permission.
The Clinical Case - Risk Distribution
A hospital makes a general-purpose clinical AI assistant available to all clinical staff. It is not mandated. Clinicians consult it at their discretion. Usage data after six months shows the predictable pattern. Use is highest among the most junior staff, and highest of all overnight.
That is entirely rational behaviour. A first-year doctor at three in the morning, facing a question outside her experience, with the on-call registrar in theatre, has a real problem and a plausible tool in front of her. Nobody is misusing anything.
But set it against the evidence. The least experienced readers are the most susceptible to following incorrect machine advice. The mechanism operates through fluency, which is exactly what makes the tool useful at three in the morning. And the alternative - calling a senior - carries a social cost that the tool does not.
So the tool's risk is not distributed the way its benefit is. It concentrates in the least experienced clinicians, at the least supervised hours, on the questions furthest outside their competence. And the deployment decision - make it available to everyone, mandate nothing - is the decision that produced that distribution. None of this appears in the system's evaluation, because the evaluation measured the system, and this is a property of the deployment.
One question for your institution
Who at your institution is responsible for the distribution of an AI tool's risk across your workforce, as distinct from its accuracy? In most organisations, nobody owns that question.
What 'Human in the Loop' Does and Does NOT Do
What it does
- Preserves clinical accountability, which matters on its own
- Catches outputs that contradict what the clinician already knows about the patient
- Catches outputs that are internally incoherent
- Provides real correction, of an unmeasured size
- Is necessary, and should not be weakened
What it does NOT do
- It does not survive fluency. A coherent, confident output engages confirmation rather than reconstruction.
- It does not hold at scale. Review quality under a full clinical load is not review quality in a study.
- It is weakest where it is most needed. The least experienced clinicians are the most susceptible, and are disproportionately the ones working unsupervised.
- It is not measured. No deployed system reports the rate at which its errors are caught by its human reviewers.
- It has no specification. Nobody has stated what a review consists of, how long it takes, or what its failure rate is.
- It does not restore the broken proxy. The reviewer is reading for coherence, which is the one thing the system produces reliably.
A human in the loop names a person, not a control. A control has a specification, a measured failure rate, and conditions under which it is known to fail. Until those exist the phrase describes an intention rather than a safeguard.
Six Questions to Ask
- Has this system been evaluated for its effect on clinician reasoning, or only for its own accuracy?
- Who uses it, when, and at what level of seniority - and does the risk profile match the usage profile?
- When the system is wrong, what does that look like? Show me examples of its errors, not its successes.
- Does its output carry any signal of uncertainty that a reader can act on?
- What is a review, specifically - what does it consist of, and how long does it take?
- Who owns the distribution of this tool's risk across our workforce?
Question 3 is the one to insist on. Every vendor can show successes. A supplier who cannot show you a representative set of their system's errors has either not looked or will not say, and both answers tell you something.
What to Reconsider
Think of a colleague whose written reasoning you trust. You have never verified most of what they have told you, and you did not need to: for as long as medicine has existed, fluent clinical reasoning could only be produced by someone who possessed it. The inference was sound every time you made it. It is now the specific inference that fails - and you have no habit of withholding it, because until recently there was nothing to withhold it from.
The assessment
The Clinical Intelligence Readiness Instrument
Everything to this point has described how clinical meaning is made, and where it is lost along the way. It does not tell you where your own institution stands.
That is not a question a book can answer. It is answered by measurement - against conditions stated in advance, and evidenced rather than asserted.
The Clinical Intelligence Readiness Instrument (CIRI™) exists for that purpose. It assesses an institution against the three conditions this guide has argued are prior to any clinical AI deployment: semantic sovereignty, clinical authorship, and infrastructural readiness. It returns a position, not an opinion.
What the assessment asks, and how it derives a score, is not set out here. Publishing the questions would change the answers.
Parts II and IV are not published here. Part II examines each standard clinically - what it carries, what it carries only when configured, and what it drops without recording that anything was dropped. Part IV takes up what a clinician can establish, and what a record will never hold. Both are part of the full guide, which will be available for purchase.