“Keeps hallucinating” is a hypothetical incident description, not yet a diagnosis. Teams often use the word for several failures: unsupported factual claims, stale retrieval, incorrect tool results, instruction conflicts, or a response that is reasonable but violates a product rule. Debugging begins by turning the complaint into a reproducible failure category.
Our chatbot keeps hallucinating. How would you debug it?
Why the common answer is incomplete
Most candidates answer: “Improve the prompt or lower the temperature.”
Those changes may reduce variability, but they do not identify the failing subsystem. A perfectly consistent model can repeatedly produce the same unsupported answer. Prompt changes can also hide retrieval bugs, stale data, or missing authorization filters. If the team cannot say which evidence reached the model and why the answer was judged wrong, prompt iteration is guesswork.
Treat answer quality like a production reliability problem with inputs, traces, classifications, and regression tests.
A production approach
1. Define a failure taxonomy
Separate unsupported claims, contradiction of provided context, missing required facts, incorrect citations, tool-use errors, stale answers, unsafe content, and formatting failures. Attach one or more labels to real incidents. The remediation differs: a citation mismatch points toward retrieval or context alignment, while an incorrect database balance may indicate a tool or freshness problem rather than model knowledge.
2. Capture a reproducible trace
Store the user input, authorized retrieval query, retrieved document identifiers and versions, ranking scores, final context, prompt version, model configuration, tool calls, tool responses, and final output. Redact sensitive values according to policy. Add a correlation ID so support reports can find the exact request. Without the original trace, a later rerun may use different documents or model behavior and fail to reproduce the incident.
3. Identify the first incorrect boundary
Walk the trace from source to response. Was the source data correct? Did retrieval include the required evidence? Did reranking discard it? Did prompt assembly truncate or bury it? Did a tool return an error encoded as a normal result? Did the model ignore available evidence? Fix the earliest incorrect boundary because downstream prompt rules cannot reliably repair missing or corrupt input.
4. Build a small adjudicated evaluation set
Turn representative failures and successful controls into a versioned dataset. Include the expected answer characteristics, required evidence, prohibited claims, and acceptable uncertainty. Use deterministic checks for citations, schema, and tool arguments where possible; use human review or carefully calibrated model grading for semantic quality. Measure by failure category rather than one blended score.
5. Improve grounding mechanics
If retrieval is the issue, revise query construction, metadata filtering, chunk boundaries, candidate count, or reranking. If context packaging is the issue, remove duplicates, preserve document titles and dates, and make source boundaries explicit. Instruct the model to distinguish supplied evidence from general knowledge and to state when evidence is insufficient. The instruction must match actual product behavior; do not demand citations if the system cannot verify them.
6. Constrain high-risk outputs
Use typed schemas for structured responses, allowlisted values for decisions, and server-side validation for tool arguments. For consequential claims, require evidence identifiers that the application can verify against the retrieved set. Route low-confidence or conflicting-evidence cases to clarification, a safer fallback, or human review. The model should not be the final authority on whether its own output is supported.
7. Separate freshness from factuality
A response can be faithful to stale context and still be wrong for the user. Record source timestamps, enforce freshness windows per domain, and invalidate derived caches when underlying content changes. For rapidly changing data, use an authoritative tool at request time rather than relying on embeddings or model memory. Display the effective data timestamp when it affects the decision.
8. Release changes through regression evaluation
Run the adjudicated set before deploying prompt, model, retrieval, or chunking changes. Compare category-level regressions, latency, and cost. Use a small production rollout with trace sampling and user feedback tied to requests. Keep the previous version available for rollback. Quality changes are software releases and need the same discipline as changes to storage or networking.
Engineering trade-offs
More retrieved context can improve recall but also introduce contradictions and dilute relevant evidence. Strict refusal rules reduce unsupported claims but may frustrate users when a useful partial answer was possible. Human review improves safety for high-impact decisions but adds delay and operational cost. Rich traces improve debugging but increase privacy and retention obligations. Model-based graders scale evaluation but can share biases with the system under test.
Choose controls based on the harm of an incorrect answer, the availability of authoritative evidence, and the cost of abstaining.
Common mistakes
- Changing the system prompt before preserving a failing trace.
- Using one “hallucination rate” that mixes unrelated failure modes.
- Evaluating only hand-written happy-path questions.
- Assuming a cited document supports the sentence without checking the passage.
- Treating lower temperature as a factuality guarantee.
- Logging sensitive prompts and documents without retention or access controls.
- Letting the model decide whether its own unsupported answer is safe to ship.
Failure handling and observability
Track unsupported-claim rate, citation validity, retrieval recall on labeled cases, refusal rate, stale-source rate, tool error rate, and user corrections. Slice metrics by feature, source collection, language, prompt version, model, and customer. Alert on sudden shifts in retrieval-empty responses, document-version mismatch, tool fallbacks, and answer-validation failures.
At runtime, preserve the trace, return calibrated uncertainty, and avoid automatic retries that merely generate a different unsupported answer. Retry only when the failure is transient, such as a timed-out retrieval service. When evidence conflicts, expose the conflict or request clarification instead of forcing a confident synthesis.
Interview insight
The interviewer is not testing whether you know a clever anti-hallucination prompt. They are testing whether you can isolate the failing boundary, create an evaluation loop, and design the product to handle uncertainty. A production answer replaces “the model made something up” with evidence about data, retrieval, generation, and validation.