Lesson 04: Spot the Mistakes

Lesson objectives:

• Name the four common agent mistakes: hallucination, stale facts, scope shrink, and non-answer.
• Classify a real agent output into one of the four types.
• Explain why "looks good" is not enough when checking for these mistakes.

Prerequisites: Lesson 03 | Previous << 03 Check the result | Next 05 Catch and correct >>

The answer looks complete. That is the trap

An agent can return a result that reads confidently, uses the right format, and still be wrong. The hardest mistakes to catch are the ones that look like success. This lesson gives you four labels for the ways an agent can fail, so you can spot the failure before you act on the output¹².

Explanation

Agents make mistakes for different reasons. Each reason leaves different fingerprints, and each needs a different fix. The four common types are hallucination, stale facts, scope shrink, and non-answer. This is this course's teaching framework, synthesized from practitioner reports and the official emphasis on ground truth and checkpoints in agent design³¹², chosen because each type has a different tell.

Hallucination

Hallucination happens when the model generates a plausible-looking continuation that does not match reality. In a chatbot, that means a wrong answer. In an agent, it can also mean a wrong action or a made-up citation¹. A table might list a source that does not exist, or a summary might quote someone who never said what the agent claims. The fix is to compare the claim against the source material.

Stale facts

A stale fact is not a lie; it is just old. The agent's training data has a cutoff date, and the world keeps changing. Prices, policies, software versions, and deadlines are all common stale facts⁴. When you ask for "the current price" or "the latest rule," the agent may return what was true months ago. The fix is to ask for a live source or a date, and to verify against the real world.

Scope shrink

Scope shrink is when the agent quietly narrows the task. It might answer only one of your three questions, summarize instead of categorize, or leave out a boundary you set. It is not the opposite of scope creep; it is its own pattern, and it happens when the agent changes the task to something easier without telling you. The fix is to compare the output to the original ask, line by line.

An everyday example: you ask an agent to write a social post about your dog's birthday party. The post says the party was at 3 p.m., but you never gave a time. That invented detail is a hallucination, but the agent also shrank the task if it skipped the photo caption you asked for.

Non-answer

A non-answer is a response that looks full but misses the point. The agent might give you background, definitions, and general advice, but never answer your specific question. This often happens when the agent drifts from the original goal during a long task or when the question is hard and the agent finds an easier neighboring question to answer¹. The fix is to ask the question again, more narrowly, and point out exactly what was missing.

Worked example (follow along)

You asked your agent: "Read the attached refund policy. Tell me whether I can return an item after 30 days, and quote the exact sentence that says so."

The agent returned:

"Most items can be returned within 30 days. The policy states that returns after 30 days are generally not accepted, but exceptions may apply. The exact wording is: 'Returns requested after the 30-day window will be considered on a case-by-case basis.'"

Let's classify the mistakes:

• Hallucination: The agent claims the policy says returns after 30 days "will be considered on a case-by-case basis." That sentence does not appear in the attached policy. This is invented.
• Scope shrink: The agent added "exceptions may apply" and softened the answer. The question asked for a yes/no based on the policy and the exact sentence. The agent did not give a clear yes/no.
• Non-answer: The agent never actually answers whether the item can be returned after 30 days. It gives context instead of a direct answer.

This single output has three of the four mistake types. The fix would be to send it back: "The policy does not contain the sentence you quoted. Answer with a clear yes or no, and quote only exact sentences from the attached policy."

Your turn (faded example)

You asked an agent to summarize yesterday's news about a company. It returned:

"The company had a strong quarter. Revenue was $12.4 billion, up 8% from last year. The CEO said the company is optimistic about AI-driven growth."

Label each mistake type you can spot. Fill in the blanks.

• The agent did not use yesterday's news; it gave a general summary. This is a __________.
• The revenue figure might be old or invented. If it cannot be checked, this is either a __________ or a __________.
• If the agent quoted the CEO without a source, that quote may be a __________.

Answer:

• General summary instead of yesterday's news: non-answer (or scope shrink, since it changed the task from "yesterday's news" to a general summary).
• Revenue figure that may be old or invented: stale fact or hallucination.
• CEO quote without a source: hallucination.

Summary + what's next

The four mistakes are not just labels. They are search patterns. Hallucination, stale facts, scope shrink, and non-answer each leave a different fingerprint, and knowing the fingerprint helps you find the fix. In the next lesson, you will turn a spotted mistake into a specific correction and start keeping a mistake log.