Short answer
A language model produces the most plausible answer rather than checking facts, so it sometimes confidently makes up incorrect information — this is called a hallucination. The risk drops when AI answers based on your documents (RAG), cites sources in its answers and, where it matters, the result is checked by the system or a person.
Why it happens
A large language model is trained to continue text plausibly. If it lacks information, it will still write an answer that sounds right. This happens especially with:
- exact figures, dates and quotes;
- narrow questions about a specific company;
- the latest information that isn’t in the model’s training data.
How to reduce errors in company solutions
- Give it sources — with RAG AI answers from your documents, not “from memory”.
- Ask it to cite the source — then a person can check quickly.
- Structured checks — for example, an invoice amount is compared with the order in the system.
- Let AI say “I don’t know” — the instructions allow and even require it.
- A person for important decisions — payments, contracts, answers with legal consequences.
Does this mean AI can’t be trusted?
No — it means AI should be used as a powerful assistant with controls. In a well-built system, AI does most of the work and a person checks only the unclear cases.