Why AI makes things up — and how to catch it

Ask an AI assistant about a historic bridge in your district and it may give you a beautiful paragraph: the king who built it, the architect who designed it, the year the arches were finished. The Class 8 lesson on this opens with exactly that situation:

The king is right. The architect never existed. The arches belong to someone else's bridge. Not one "perhaps".

This has a name. It is called a hallucination, and anyone who uses AI tools needs to understand it.

What a hallucination is

The book's definition:

A hallucination is a confident, fluent statement that is simply false. It comes from how the model works; it can't be switched off.

That second sentence is the important one. A hallucination is not a bug that the next update will fix. It is a side effect of what these tools are.

Why it happens

A chatbot is a generative model. Where a classifier asks which class is this?, a generative model asks what comes next? — and it answers by predicting likely words, one after another, based on the patterns in everything it was trained on.

The book puts it in three short lines:

• The model predicts likely words. It doesn't look anything up or compare with the world.
• It sounds sure of everything because it learned the shape of factual writing.
• It's worse on local questions — little is written about our villages and shrines.

So when a model doesn't "know" something, it does not go quiet. It produces the words that would typically appear in an answer like that. A real king, an invented architect, arches borrowed from a more famous bridge — all assembled into text that sounds exactly like the truth.

Why it matters more here

For a question about Paris or the Pythagorean theorem, there is so much text in the training data that the likely words are usually the right ones. For a question about a shrine in Baramulla, a Sopore orchard's history or a Kashmiri proverb, there is very little — and the gaps get filled with fluent guesses. (This is one reason Kashmiri needs its own AI.)

The local question is exactly where people are most likely to trust the answer, because it is about something they half-know already.

A routine for catching it

You don't need to stop using AI. You need a habit. The lesson's rule is six words: "Fluent is not the same as true. No source, no sentence." In practice:

1. Separate the claims. Break the answer into individual facts — names, dates, numbers, places.
2. Find a named source for each. A textbook, an official site, a person who knows. Not "the internet says".
3. Open the source before you repeat the claim. A link next to a sentence looks like proof. It is only proof once you have read the page — that is one of the trust rules in our AI tools course.
4. Mark each claim true, false or can't tell. "Can't tell" is a real answer. Leave unverified claims out.
5. Watch numbers and names hardest. Those are where invented details hide.

The scholar's answer

The Class 8 book reminds students that saying I do not know was once the mark of a serious scholar. Imam Mālik ibn Anas is remembered for answering question after question with lā adrī — "I do not know" — rather than filling the gap. As the book says: a language model has no lā adrī setting. You have to supply it.

That is the whole skill. Let the machine be fluent. Be the one who checks.