Why AI answers make things up, and how sources help you check them

Confident and wrong

Anyone who uses AI chat tools for long enough runs into it: an answer that reads well, sounds sure of itself, and turns out to be false. A quote nobody said, a paper that doesn't exist, a date that's off by a year. Researchers usually call this hallucination. The US National Institute of Standards and Technology prefers the word confabulation, which it defines as "the production of confidently stated but erroneous or false content" by which users may be misled (NIST AI 600-1, 2024).

It has real consequences. In 2023 a US federal court sanctioned lawyers who had submitted court decisions that didn't exist, generated with an AI tool, and who kept standing by them after the court questioned them (Mata v. Avianca, opinion and order on sanctions). The same order notes there is nothing inherently improper about using a reliable AI tool for assistance. The failure was not checking.

Why it happens

A large language model writes by predicting likely text, one piece at a time, based on patterns learned from its training data. Most of the time, likely text is also true text. When the model lacks the information, though, it can still produce something that has the shape of an answer. One major survey defines hallucination in these models as generating plausible yet nonfactual content (Huang et al., 2025), and an earlier survey found the problem across summarization, dialogue, question answering and translation (Ji et al., 2023).

A 2025 paper adds an incentive problem. It argues that language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty (Kalai et al., 2025). On a test that scores "I don't know" as zero, a confident guess scores better on average, so systems are pushed toward guessing.

What retrieval changes

One common fix is to give the model the right material before it answers. Retrieval-augmented generation, or RAG, first searches a set of documents for relevant passages and then asks the model to answer using them. In the paper that introduced the term, models with retrieval produced more specific, diverse and factual language than a model answering from its training alone (Lewis et al., 2020).

Retrieval narrows the problem but doesn't remove it. The search can miss the passage that matters, the documents can be wrong or out of date, and the model can still add claims the passages don't support. The same 2020 paper lists providing provenance for answers as an open problem.

A citation is a pointer, not proof

Citations make an answer checkable, which is valuable. They don't make it correct. When researchers audited four AI search engines in 2023, they found that on average "a mere 51.5% of generated sentences are fully supported by citations and only 74.5% of citations support their associated sentence" (Liu, Zhang & Liang, 2023).

So the value of a citation depends on whether you follow it. A source that opens the exact passage takes seconds to check. A source that opens a 40-page PDF asks a lot more of you.

How to check an AI answer in about a minute

1. 1Find the claim that matters. Most answers have one or two sentences you would actually act on. Check those, not every word.
2. 2Open the source for that claim. If there is no source, treat the answer as a lead to verify, not a fact.
3. 3Find the actual sentence. Does it say what the answer says, or only something on the same topic?
4. 4Check the context: the date, who said it, and whether a later source replaces it.
5. 5If the sources don't cover your question, believe that. An honest "not in these sources" is more useful than a guess.

It also helps to give the tool less room to guess. Point it at the specific documents that should contain the answer, and ask it to say when they don't.

How we approach it at Lumyo

These ideas shaped how Lumyo works. You choose which notes a chat may read, answers cite the notes they used, and tapping a source opens the note at the exact line. When your chosen notes don't cover a question, Lumyo is built to tell you so instead of guessing.

That still doesn't make every answer right, which is why the source is one tap away: a list of sources shows where the retrieved content came from, and reading the cited line is how you check that a sentence is actually supported.