The first time a language model handed me a citation that didn't exist, I assumed it was a bug. The format was immaculate. The journal was real. The authors were real people who actually work in the field. The DOI was correctly shaped, down to the registrant prefix. The only problem was that the paper on the other end of it had never been written.
I've since learned this isn't the model malfunctioning. It's the model working exactly as designed. A fabricated citation is one of the purest expressions of what a language model is — and understanding why is a small, clarifying lesson in how these systems actually work.
It is not rare, and it is not a rounding error
"AI sometimes makes up sources" undersells the scale. A 2023 study in Scientific Reports audited the bibliographies ChatGPT produced: with GPT-3.5, 55% of citations were entirely fabricated. The jump to GPT-4 cut that to 18% — better, but "fewer than one in five of your sources is fake" is not a foundation you build scholarship on. And of the citations that did point to real papers, roughly a quarter still carried substantive errors — wrong year, wrong pages, wrong DOI.
A separate study of ChatGPT-generated medical content was worse: of 115 references, 47% were fabricated, 46% were real but inaccurate, and 7% were both real and correct. Across several such studies, roughly half of all citations were invented outright.
And when this escapes the lab, it gets expensive. In 2023, two New York lawyers filed a brief built on cases ChatGPT invented — fictitious airlines, fabricated quotes, citations to opinions that did not exist. When the court asked for copies, the model doubled down, assuring them the cases were real and could "be found on Westlaw and LexisNexis." A judge called one summary "gibberish" and fined them $5,000. The model didn't just get it wrong. It got it wrong with references.
Why a citation is the easiest thing in the world to fake
Here's the mechanism, and it's worth being precise. A language model generates one token at a time, always reaching for the most probable next piece given everything before it. It is, at bottom, a machine for plausible continuations.
Now look at a citation as a string: Lastname, F. M. (Year). Title of the paper. Journal Name, Vol(Issue), pp–pp. It is one of the most structured, most repeated, lowest-entropy patterns in the entire training corpus. The model has seen that shape millions of times and can reproduce it flawlessly — the comma after the surname, the parenthetical year, the journal rhythm, the page range.
But the shape is the only part it has truly learned. The binding between that shape and a specific real paper is not something it can look up while generating — there is no database behind the curtain. So it does what it always does: fills the slots with the most probable contents. A plausible author for the topic. A plausible journal. A plausible year. A plausible-looking DOI. Each slot is locally probable; the whole is fiction. The fluency isn't hiding the problem — the fluency is the problem. A clumsy fake you'd catch. A perfect one you cite.
The DOI is the tell
If you want the seam to show, look at the DOI. A DOI is engineered to be the opposite of vague: a permanent, machine-resolvable address — a 10. prefix, a registrant code, and a suffix that resolves through the global DOI system to exactly one work. It either resolves or it doesn't.
But a model doesn't see an identifier. It sees tokens — a string like 10.1038/s41586-… gets shredded into fragments, and the model predicts plausible next characters, not a real address. The result is a DOI that is syntactically valid and semantically empty: correctly formed, pointing at nothing. This is why a fabricated citation survives a human skim and dies the instant you resolve it. The eye accepts the shape; the resolver checks the referent. They are not the same test.
(If you read my piece on tokenization, this will feel familiar. The model that "spells before it reads" a word is the same model that "formats before it knows" a citation.)
Memorized vs. interpolated — and why it fails where you need it most
There's a pattern hiding in the numbers. The model fabricates obscure papers far more than famous ones, and GPT-4 fabricates less than GPT-3.5. Same cause: a citation that appeared thousands of times in training is effectively memorized — the model can reproduce it because it has, in a sense, stored it. A citation from the long tail — recent, niche, specific — was never stored, so it gets interpolated: invented from the surrounding probability.
Which means the tool is most reliable exactly where you least need it — citing the canonical paper everyone already knows — and least reliable exactly where you most need it: the specific, recent, narrow source that would actually strengthen your argument. Reliability runs inversely to usefulness. That isn't a tuning problem you patch next version. It's the shape of the thing.
The category error underneath all of it
Strip it back and the mistake is conceptual: we keep asking a probability engine to perform a database lookup. Generating fluent text and retrieving a verified fact are different operations, and no amount of prompting collapses them into one. "Only cite real papers" cannot work, because at generation time the model has nothing to check against — no ground truth, only priors.
The fix isn't a better prompt. It's architectural: separate generation from verification. Let a model draft, but resolve every reference against an authoritative index — CrossRef, PubMed, the DOI system itself — before a single one is allowed to count. A DOI either resolves to the claimed paper or it does not. A citation you generated is a guess in a tuxedo. A citation you retrieved is a fact. Treating those two as interchangeable is the whole problem; keeping them apart is the whole solution.
The part worth sitting with
What unsettles me isn't that AI gets sources wrong — tools have always had failure modes. It's that this failure mode is dressed for success: perfectly formatted, supremely confident, optimized to pass exactly the kind of quick review most citations ever get. It fails in the one place nobody looks.
So a question for anyone who has pasted an AI-generated reference into a document: did you resolve the DOI? Not skim the title — actually follow it to the paper. Because if you didn't, you don't have a source. You have a sentence wearing one.
Sources: Walters, W. H., & Wilder, E. I. (2023). "Fabrication and errors in the bibliographic citations generated by ChatGPT." Scientific Reports, 13, 14045. · Bhattacharyya, M., Miller, V. M., Bhattacharyya, D., & Miller, L. E. (2023). "High rates of fabricated and inaccurate references in ChatGPT-generated medical content." Cureus, 15(5), e39238. · Mata v. Avianca, Inc., No. 22-cv-1461 (S.D.N.Y. 2023).
The DOI fragment 10.1038/s41586-… above is a format illustration, not a reference to a real paper — fitting, given the subject.