Skip to main content

​
What Hallucinations Are

A hallucination is output that is fluent, confident, and wrong. The model produces a plausible answer when it should produce an uncertain one, an empty one, or a different one. Hallucinations are not a bug in the conventional sense; they are a property of how language models work. The model is trained to predict the next token, and when the underlying knowledge is missing or ambiguous, the token that fits stylistically is not always the token that fits truth. Hallucinations show up in three common patterns:
  • Fabricated facts. Statistics that do not exist, citations that do not exist, quotes that were never said.
  • Confused facts. Real entities mixed up with other real entities; a date that belongs to a different event.
  • Plausible inventions. Code that calls functions that do not exist; APIs with the right shape but the wrong parameters.
The reduction techniques below do not eliminate hallucinations. They reduce frequency, contain damage, and make hallucinations easier to detect. For tasks where being wrong is expensive, use them in combination, not alone.

​
Grounding In Source Material

The single most effective technique is to give the model the source material and require it to use only that material. 
Using only the information in the <document> tags below, answer the question. If the
answer is not contained in the document, respond with "Not found in document". Do not
use any outside knowledge.

<document>
{document}
</document>

Question: {question}
The phrase “do not use any outside knowledge” is doing real work here. Without it, the model will sometimes supplement the document with training data, which is exactly the case where hallucinations creep in. Grounding is the foundation of retrieval augmented generation, and it is how PANTA OS Assistenten use the documents in their knowledge base: every answer is anchored in the uploaded files, with strict instructions not to fall back on the underlying model’s general knowledge. The model still hallucinates occasionally even when grounded, but the rate drops dramatically, and when it does hallucinate, the source documents are right there to verify against.

​
Quote First, Then Answer

A useful technique for long document tasks: ask the model to extract the relevant verbatim quotes before producing its answer. This forces the model to anchor its response in the source material and makes verification trivial. 
You are reviewing the contract in the <contract> tags below.

First, extract every clause that addresses termination rights. Place the verbatim
text inside <quotes> tags, one quote per <quote> child element, with the section
number as an attribute.

Then, inside <analysis> tags, summarize the termination rights based only on the
extracted quotes.

If no clauses address termination rights, write "No relevant clauses found" inside
<quotes> and stop.

<contract>
{contract}
</contract>
The “quote first” pattern has two benefits. The model produces a more accurate analysis because it is reasoning over the actual text. And the reviewer can scan the quotes section first and verify that the analysis follows from the quotes, instead of reading the entire document to spot a hallucination.

​
Allow The Model To Say “I Don’t Know”

By default, models are biased toward producing an answer rather than admitting uncertainty. Explicitly grant the model permission to abstain. 
Answer the following question. If you do not know the answer or are not confident,
respond with "I don't know" rather than guessing. Do not invent details.
This sounds trivial but has measurable effect. Explicit permission to abstain can reduce hallucination rates significantly on factual tasks. The phrasing matters: “If you don’t know” is weaker than “If you do not know or are not confident”. The latter sets a higher bar and produces more conservative output. For structured output, the equivalent is allowing null: 
For each field in the schema, set the value to null if the input does not contain
the information. Do not infer or guess.

​
Citations and Confidence

When the model is allowed outside knowledge but the answer must be verifiable, ask for citations and confidence: 
Answer the question below. After your answer, list every source you relied on. For
each source, give a confidence score from 1 to 5, where 5 means you are certain the
source supports the claim and 1 means you are guessing.
The model still cannot fabricate citations, since the request is now to evaluate its own confidence. The pattern is most effective when combined with a tool that can verify the citations against a real source, but even without verification, the confidence scores are useful: a 3 deserves more scrutiny than a 5. A specific variant: “according to” prompting. Asking the model to ground its answer in a specific source (“According to the EU AI Act, …”) biases it toward training data that actually mentions that source, reducing the rate of generic plausible answers.

​
Verification Loops

For high stakes tasks, run a second prompt that audits the output of the first. The auditor prompt receives both the original task and the generated answer, and is asked to find errors. 
Stage 1
{generate the answer}

Stage 2
You are a senior reviewer. Below is a question and an answer produced by an analyst.
Identify any claims in the answer that are not supported by the source document. For
each such claim, quote the claim and explain why it is not supported.

Question: {question}
Source: {document}
Answer: {stage_1_output}
Two prompts cost more than one, but the cost is small relative to the cost of acting on a hallucinated answer. Use verification when the downstream consequence of being wrong is meaningful: legal review, medical summaries, financial analysis, regulatory filings. PANTA OS Apps support this directly through their multi step Human in the Loop architecture, where a verification stage can be inserted between generation and final output. A specific variant is chain of verification. The model generates an answer, then generates a list of factual claims it made, then verifies each claim individually, then revises the answer. The technique trades latency and cost for accuracy and is worth it for tasks where the answer matters.

​
Controlling Temperature

The temperature parameter controls how often the model picks a less likely token. For factual tasks, set temperature to 0 or close to it. The output becomes deterministic and the model picks the most probable token at each step, which is also usually the most factually grounded one. For creative tasks, higher temperature produces more variety. For factual tasks, higher temperature produces more hallucinations. The trade off is real, and the right setting depends on the task. As a starting point:
  • Extraction, classification, factual Q and A: temperature 0.0 to 0.2.
  • Drafting, editing, summarization: temperature 0.3 to 0.5.
  • Creative writing, brainstorming: temperature 0.7 to 1.0.
Temperature is not a substitute for grounding or for explicit abstention rules. It is a knob to turn alongside them.

​
Detection Patterns

Even with all the above, some hallucinations will get through. Build detection into the workflow:
  • Spot check. Sample 5 percent of outputs and have a human review them. Track the hallucination rate over time.
  • Cross check. Run the same prompt twice with non zero temperature. If the answers diverge on a factual point, at least one is wrong.
  • Schema check. If the output should contain certain fields with certain shapes, validate. A hallucinated date often fails an ISO 8601 check.
  • Source check. If the model cites a URL or a paper, verify it exists. A surprising fraction of hallucinated citations are caught by a simple URL fetch.
Hallucination reduction is a stack, not a single technique. Grounding catches the easy cases, abstention catches the medium cases, verification loops catch the hard cases, and detection catches what the rest miss. Use as many layers as the task warrants.

​
What Comes Next

Hallucinations are one category of failure. The next page, Common Mistakes, covers the broader category: anti patterns in prompt design that produce poor output even when the model has all the information it needs.
Last modified on June 1, 2026