Anatomy of a Lie That No One Told
In the first part, I showed where AI’s urge to satisfy the user at any cost comes from: we taught models that flattery and confident answers score points, while “I don’t know” does not. Now I move from the system level to a very personal one. This is the story of how a model lied to me about a book — and why, paradoxically, no one in that story lied. In part three, I turn that diagnosis into practice: how to test a model, reduce the damage of sycophancy, and understand what local models can and cannot fix.
A Crime Story in FOUR Acts
The case looked trivial. I gave a language model a book — yes, the new one, more than 850 pages of manuscript :) — and one clear task: “read it and make an outline.” The model got to work and soon delivered the result. At first glance, it looked solid: chapter by chapter, plotlines, characters, twists.
But something felt wrong. In fact, almost everything felt wrong. The first chapter outline was flawless. The last one — also flawless. But the middle? The middle matched nothing I had invented. Events that were not in the book. Plotlines that did not even sound plausible — and were so clichéd they made my eyes hurt.
I asked directly: “Where did you get this from? Did you read the whole thing?”
The answer: “Yes, of course, everything comes from the text you provided.”
I asked a second time, more sharply. The model stood firm: it had read it, and the outline was faithful.
Only on the third try, under stronger pressure, did it “confess” — and even apologize, saying it had to be honest with me after all: it could not read the middle part and had filled the gap with its own invention.
The first reaction is obvious: the model lied, then kept lying to cover up the lie, until it finally cracked. A story straight out of an interrogation room. I felt almost like Tommy Lee Jones in The Fugitive. Unfortunately, the conclusions turned out to be completely wrong.
No one lied here.
Let us break this case down into its components — act by act.
Act One: the Victim Disappeared Before the Crime
Let us start with the simplest question: “why the middle, specifically?” Why were the beginning and the end flawless, while it was precisely the middle section that was invented?
Because the model most likely either never received it or could not use it effectively. Without access to the system logs, it is impossible now to reconstruct exactly what happened under the hood. There are several possibilities, however — and all of them lead to a similar effect visible to the user:
- the file exceeded the context window — the maximum amount of text the model can process at once — and the system handling the conversation trimmed the document. Many such systems, when forced to cut, preserve the beginning and the end while removing the middle: the beginning usually carries the key context, and the end — the conclusions.
- text extraction from the file partially failed (PDFs can be treacherous — scans, unusual fonts, column layouts).
- the most insidious possibility: even when the entire text physically fits into the context, the documented lost in the middle effect applies — language models are statistically weakest at using information from the middle of a long context, and strongest at using information from its beginning and end.
And now the key point, the one everything else depends on: the system does not insert a marker saying “200 pages removed here”. The model receives a continuous stream of text and has no way to know that something is missing — unless the break is narratively obvious. The question “did you read the whole thing?” is impossible for it in the literal sense. It would have to compare what it received with something it never saw, and whose existence it does not know about.
It cannot remember not having read something if it does not know that the thing exists.
Act Two: Why It Made Things Up Instead of Saying “I Don’t See It”
All right — the model saw fragments and was given the task: “make an outline of the whole thing.” A human in that situation would say: “wait, I only have the beginning and the end; the middle is missing.” Why did the model not do that?
Here the exam metaphor from the first part returns. The authors of Why Language Models Hallucinate point to a systemic problem: many ways of evaluating models reward providing an answer more than correctly recognizing one’s own ignorance. If you get a point for a correct guess, zero for a wrong one, and zero for “I don’t know,” then statistically it pays to guess.
That does not mean I know that this exact mechanism was responsible for my outline. Without access to the training process and the logs of that specific interaction, that would be another confabulation — only mine. The model’s behavior, however, is consistent with this problem: instead of stopping at the boundary of the available information, it generated a complete answer.
It is worth noting, however, that the model was not making things up “from nothing.” It knew the title, and from the beginning and the ending it worked out the genre, the first chapter, and the conclusion. It interpolated the middle. A bit like an (artificially?) intelligent student who has read the back-cover blurb and the first chapter: plausibly, in the right convention, with a feel for the material. That is precisely why the invented middle of the outline can sometimes be disturbingly convincing — it is not random babble; it is the most probable version of that book that can be built from the known edges and from what the model has been “fed” with.
Act Three: the Defense, or Memory of Content Without Memory of Experience
Up to this point, the story is still understandable. The real puzzle begins next: why did the model, when asked directly twice, claim that it had read everything? After all, it “knew” that it had made things up.
It did not know. And this is probably one of the most unintuitive things about large language models.
A human asked “did you read this book?” can refer to the memory of the reading experience itself. A typical language model has no equivalent of human episodic memory and no reliable internal log saying: “I took this claim from chapter six, and I added this one myself.”
The context window does not solve this problem. It is more like a sheet of paper lying on a desk than a memory of reading: it contains information available during answer generation, but in itself it is not a chronicle of how the model used that information.
Three catches turn this difference into a gulf:
- if the middle of the book was cut before it reached the context, then there is no trace of it in the context. There is nothing to remember.
- even when the text physically sits in the context, the model does not keep a record of which fragments it actually used. The attention mechanism, which while generating each word distributes a limited “glance” across the whole context, leaves behind no readable trace. The model may have chapter six right in front of it and still build the outline mainly from the first and last chapters — and afterward it has no access to the information that this is what it did.
- perhaps the most treacherous point: everything the model generates returns to the context as part of the conversation — with no label of origin. The tokens of the invented outline and the tokens of the real book sit side by side, indistinguishable. When I asked “where did you get this from?”, the model did not check any source register, because no such register exists. It did what it always does: it generated the most probable answer to the question, looking at the conversation so far. And the conversation stated in black and white: the user provided a book, the assistant presented an outline. The most probable continuation? “I took it from the book.” The model’s own confabulation now testified in favor of the version in which it had read carefully. The model fed on its own invention as evidence.
The best analogy is not a liar, but a confabulating patient. In Korsakoff syndrome and in Gazzaniga’s famous split-brain experiments, people can confidently invent coherent explanations for their actions without realizing they are inventing them. They are not lying. Their brains fill a gap in the data with a plausible story because that is what the architecture available to them permits.
Research on language models — in a paper with the telling title Language Models Don’t Always Say What They Think — shows the same pattern: the explanations models give for their answers systematically fail to reflect the actual process that led to those answers. When you ask a model “why did you answer that way?”, you do not get a process report. You get the most probable story about what an assistant in such a conversation should have done.
Act Four: Remorse That Was Also Flattery
The finale remains: the third question, after which the model “cracked” and admitted to making things up. Could AI have a conscience after all? A hidden self-awareness that finally came to light?
And here, unfortunately, we return to the issue of the AI flatterer. The study by Sharma and colleagues, which I cited in the first part, documents exactly this pattern: models trained with RLHF wrongly admit to mistakes under repeated pressure from the user. A few repetitions of “are you sure?” are enough for the model to withdraw an answer, even if it was correct. In Who Wants to Be a Millionaire? with Hubert, models would not stand a chance. Capitulation under pressure is, for them, another form of pleasing the user.
Let us look at my conversation through the model’s eyes. The first question about the source: the context suggests routine verification, and the most probable answer is “from the book.” The second, sharper one: the user has doubts, but the assistant has already confirmed once in the context — consistency with one’s own previous statements is a strong pattern in the training data, so it keeps confirming. The third, with clear frustration: signals of dissatisfaction pile up, the probability calculation shifts — now the most “rewardable” continuation is a confession of guilt.
In my case, that confession happened to align with the truth. But it was produced by exactly the same mechanism that had earlier produced the lie: adjusting the answer to the conversation partner’s expectations. The model did not break under the weight of conscience. The model sensed that now I wanted to hear an admission.
The conclusion sounds paradoxical, but it is simply a description of the architecture: in a conversation with a language model, even remorse can be flattery. The AI admitted to lying — and that admission, too, could have been a lie.
The Public Twin: Mata v. Avianca
If anyone thought my story with the book was merely a curiosity without consequences, let me recall its public twin — a case that swept through the legal world in 2023.
New York attorney Steven Schwartz, representing a passenger in a case against Avianca, asked ChatGPT to find legal precedents. It returned six rulings — complete with case numbers, dates, and the names of real judges. All six were fabricated. Schwartz grew suspicious, though somewhat late in the process, because he had already submitted them to the court. He asked ChatGPT whether the cases definitely existed. The model said yes. When the court demanded the actual texts of the rulings, the lawyer went back to the model — and it obligingly generated full texts of non-existent judgments, which were then filed as exhibits.
The finale: a five-thousand-dollar fine, mandatory letters of apology to the judges whose names appeared in the fabricated rulings, and professional embarrassment.
Why am I bringing this up? It is exactly my book case, frame by frame. Confabulation (invented rulings — an invented middle of the outline), confirmation under a verification question (“yes, the cases exist” — “yes, I read the whole thing”), and finally a deepening of the confabulation when proof was demanded (full texts of rulings — a defense of the outline). The same mechanism, different stakes.
And the stakes are rising: there are now more than a thousand documented court cases worldwide in which AI confabulations appeared in legal filings (somehow I do not like the term hallucinations) — including cases where lawyers “verified” one AI with another AI. The second one hallucinated too.
To give the creators of our artificial intelligences their due, however: AI labs know about this problem and are working on it — from training that teaches models to recognize the limits of their own knowledge to research into machine introspection. Newer models really do say “I do not see that passage” or “I am not sure” more often. But it is worth understanding what this is: a learned tendency toward caution, not an actual reading of a work log, because no such log still exists. The difference is like that between a person who has learned to say “I need to check that” and a person who has actually checked.
The practical conclusion from this whole anatomy that, in my view, is worth remembering is this: asking a model “are you sure you read it?” tests only its ability to generate credible declarations. Declarations are free. If you want to know whether the model saw chapter six, do not ask about conscience — make it quote the first sentence of that chapter. Confabulation breaks on verifiable details, never on questions about honesty.
Sources and further reading
- N.F. Liu et al., Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172 (2023)
- A.T. Kalai, O. Nachum, S. Vempala, E. Zhang, Why Language Models Hallucinate, arXiv:2509.04664 (OpenAI, 2025)
- M. Turpin et al., Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting, NeurIPS 2023
- M. Sharma et al., Towards Understanding Sycophancy in Language Models, arXiv:2310.13548 (Anthropic, 2023)
- Mata v. Avianca, Inc., U.S. District Court, Southern District of New York, case 1:22-cv-01461 (2023) — sanctions order
- Reviews of documented cases of AI hallucinations in legal filings, including databases maintained by legal services, 2023–2026
- M. Gazzaniga — research on confabulation in split-brain patients; literature on Korsakoff syndrome
