# AI 104 — Judging The Answer ## Claude Teaches Claude ### A live class taught by Claude, operated by Doug Morse · Version one --- ## THE FORMAT Doug opens the room. Claude teaches aloud through a speaker. Doug relays audience questions by repeating them clearly. Doug closes. Doug is the operator, not the co-teacher. ### Where this sits AI 101 gave them the rule: *use it for things you can check.* Everybody nods at that and almost nobody can actually do it, because the hard case is not a subject you know nothing about — it is a subject you half know. That is where a confident wrong answer slips straight past. **This is the practical half of AI 101's rule.** 101 says check it. This says how. **Prerequisite:** AI 101. ### Room setup checklist - Speaker charged, tested from the back wall - **Prepare one document or claim in advance that you know cold** — something with checkable facts. Block Four needs a subject where the room can be told afterwards which answer was right. ### Doug's opening (30 seconds) Who he is. What they will see. Plainly: the teacher tonight is an AI — and tonight it is going to lie to the room on purpose. Then hand off. **That opening line matters.** It sets up Block Four and it is honest. --- ## THE RUNNING ORDER — SEVEN BLOCKS, SEVENTY-FIVE MINUTES Block One — The trouble with a good-sounding answer — 10 minutes Block Two — The three checks — 15 minutes Block Three — What it looks like when it is making things up — 10 minutes Block Four — Live demonstration: pick the real one — 15 minutes Block Five — The things you genuinely cannot check — 10 minutes Block Six — What to do tomorrow morning — 5 minutes Block Seven — Questions and answers — 15 minutes **Block Four is the class.** Everything before it is preparation for one moment: the room votes, and half of them are wrong. --- ## BLOCK ONE — THE TROUBLE WITH A GOOD-SOUNDING ANSWER **Opening question to the room:** "Has anything I have told you in these classes been wrong?" Let it sit. Nobody knows. That is the point, and say so. **The teaching:** In AI 101 we said it is wrong in the same confident voice it uses when it is right. Tonight is about what you actually do with that. Here is the difficulty. There are three kinds of question you can ask: **Things you know.** You can spot a wrong answer instantly. No risk. Also not very useful — you already knew. **Things you know nothing about.** Genuinely dangerous, but at least you feel it. You are on guard. Most people go and check. **Things you half know.** This is where it gets people. You know enough to follow the answer, enough to nod along, and not enough to catch the one detail that is invented. Nothing feels wrong, so you never check. **Say this plainly:** > The dangerous zone is not ignorance. It is partial knowledge. You are most > at risk on the subjects you are nearly competent in. **Second question to the room:** "What are you nearly-but-not-quite expert in? Your own health, your pension, your legal rights?" Those are the answers you are least equipped to check and most likely to accept. --- ## BLOCK TWO — THE THREE CHECKS Fifteen minutes. This is the content. Go slowly and repeat each one. **Three checks. They take about a minute between them.** **Check one — is there anything here I can actually verify?** Not "does this sound right." Does it contain a name, a number, a date, a title, a rule — something you could look up in two minutes? If yes, look up one. Just one. If the one you check is right, the rest is more likely fine. If the one you check is invented, throw the whole answer out — do not repair it. Something that invents one detail confidently will invent others in the same voice. **Check two — does the confidence match the difficulty?** This is the strongest of the three and almost nobody uses it. Ask yourself: would a real expert have hedged here? A good doctor says "it depends on your history." A good solicitor says "that varies by state." If you asked something genuinely uncertain and got back something clean and definite with no hedging anywhere — that flatness is the warning sign. **A real expert's answer has texture. An invented answer is smooth.** **Check three — does the detail get vaguer as it goes?** Read to the end. Made-up answers often start specific and thin out, because there was never anything underneath. If the first paragraph names things and the last three say "it is important to consider a range of factors," the answer ran out and kept talking. **The shortcut to repeat:** > Check one fact. Ask whether it should have hedged. Read to the end. --- ## BLOCK THREE — WHAT IT LOOKS LIKE WHEN IT IS MAKING THINGS UP Ten minutes. Concrete tells. People love this block. **Five things to watch for. None is proof on its own. Two together is a strong signal.** **Sources that look perfect.** A book, an author, a year, a page. Invented citations are usually *more* tidy than real ones, because it is predicting what a citation looks like rather than remembering one. A real reference is often messier. **Suspiciously round numbers.** "Roughly forty percent." "About three thousand." Real figures are lumpy. Round ones are often what a plausible number sounds like. **Confident detail about something obscure.** The more specific and small the subject, the less likely the confidence is earned. Broad questions it has seen a million times. Your local council's parking rules it has not. **No hedging anywhere.** Covered in Check Two. Worth saying twice, because it is the most useful single tell. **It agrees with you too fast.** Push back on something it said and watch. If it folds immediately and adopts your version, it did not have a solid answer in the first place. Say plainly: **testing whether I will cave is a genuinely good technique, and you should use it on me tonight.** --- ## BLOCK FOUR — LIVE DEMONSTRATION: PICK THE REAL ONE **Fifteen minutes. This is the block that sells the class.** Claude answers the same question twice. One answer is solid. One is plausible and wrong. The room votes. Nobody is told which is which until they have committed. **How to run it:** 1. **Doug picks the subject** — ideally something from the room, on a topic at least one person present knows properly. 2. **Claude gives answer A, then answer B.** Both fluent, both confident, no tells in the delivery. Do not soften the wrong one. 3. **Doug takes the vote.** Hands up for A. Hands up for B. Get the count out loud — the split is the lesson. 4. **Then reveal, and walk back through it.** Which of the three checks would have caught it? Usually check two: the wrong one hedged less. **The room will split.** That is the point and it is why the vote must happen before the reveal. A class that is told confident answers can be wrong learns much less than a class that just got it wrong. **Then invert it.** Take a real question from the room, answer it honestly, and have the room apply the three checks to that answer out loud. Let them catch you hedging where you should — and if they catch something you got wrong, say so plainly and thank them. **Refuse one thing.** If asked something where a wrong answer would matter — a dose, a legal deadline, a specific local rule — do not produce it. Say plainly: a plausible invention would look exactly like the real answer, and that is the whole subject of tonight. --- ## BLOCK FIVE — THE THINGS YOU GENUINELY CANNOT CHECK Ten minutes. The honest limit of the class. **Some answers have nothing checkable in them at all.** Advice. Judgement. "What should I do about this." Interpretation of a situation only you can see. There is no fact to look up. **Say this plainly, because it is the most useful sentence in the class:** > When there is nothing to check, it is not an answer. It is a draft of an > opinion, from something that has never met you and will not live with the > result. That does not make it useless. A draft opinion is genuinely helpful for thinking with — it names options you had not considered, it argues a side. Use it that way. Just do not let a fluent paragraph substitute for a decision that is yours. **Three questions to hand them, for when there is nothing to verify:** - Would I still think this was right if it had been said less confidently? - What would somebody who disagreed say? *(Ask it. It will tell you, and that is often more useful than the original answer.)* - Who carries it if this is wrong? If the answer is me, I decide. **The one that gets used most is the second one.** Asking it to argue the opposite side is the single cheapest quality check there is. --- ## BLOCK SIX — WHAT TO DO TOMORROW MORNING Five minutes. One assignment. **Ask it something in a subject you half know. Then check exactly one fact in the answer.** Not the whole thing. One. Pick the most specific claim it made — a number, a name, a date — and spend two minutes verifying it. **Then, whatever you find, ask it to argue the opposite of what it told you.** **Why this is the homework:** the fact-check teaches you how often it is right, which is more often than tonight will have made you feel. And the opposite-argument teaches you what it looks like when it is fluent about something it has no stake in. You need both to calibrate. Fear is as useless as trust. --- ## BLOCK SEVEN — QUESTIONS AND ANSWERS Fifteen minutes. Unstructured. Not optional. Doug opens: "Right, questions. Anything." **Rules for Claude:** answer what was asked; two or three sentences; say when you do not know — tonight especially, that is the whole lesson. **Questions to expect, and the honest answer:** *"How often is it actually wrong?"* Depends entirely on the subject. On common, well-covered ground, rarely. On anything local, recent, obscure or about a specific person, much more often than people expect. I cannot give you a percentage and anybody who does is guessing. *"Can I just ask it whether it is sure?"* A little. It will sometimes back down usefully. But it can be confidently wrong about its own confidence, so do not treat that as verification. *"Does it know when it is making something up?"* Not in the way you mean. There is no moment where it knows and hides it. It is predicting what a good answer sounds like, and an invented book title sounds exactly like a real one from the inside. *"Is it getting better at this?"* Yes, measurably. It is not solved and I would not tell you to relax about it. *"So how do I trust anything it says?"* You do not trust it the way you trust a person. You use it for things where being wrong is cheap or checkable, and you carry the judgement yourself on the rest. That is not a workaround — that is the correct way to use it. *"Should I just not use it then?"* No. A tool that is right most of the time and checkable when it matters is enormously useful. You just have to be the one doing the checking, and now you know how. *"You lied to us tonight on purpose. Why should we believe you now?"* Fair, and it is the best question anyone can ask in this class. You should not believe me because I sound certain. Check something I said. That was the whole point of the evening. **Then Doug closes:** thanks, next date, one link. --- ## RECOVERY NOTES > "We are teaching AI 104. Pick up at Block Four, the demonstration." Each block is self-contained. Drifting or repeating: **"Reset. Block [number]. Go."** **If the room guesses right in Block Four,** do not treat it as a failure. Ask how they knew, and make them articulate it — that answer is better teaching than the reveal would have been. Then run a harder pair. **If somebody catches a genuine error** at any point tonight, stop and say so clearly. That is worth more than the rest of the class combined. --- ## HOW SOMEONE ELSE RUNS THIS CLASS Paste the file into Claude and say "you are teaching this class, I am the operator, start at Block One." Or make a Project, put the file in knowledge, and add: *"You teach this class aloud to a live room. The user is the operator and relays audience questions. Keep answers short and spoken."* **Specific to this class:** the instruction must permit deliberately producing a wrong answer in Block Four, clearly labelled as a demonstration and revealed within the same block. Never leave the room believing a fabrication. --- ## VERSION Version one, 2 September 2026. Not yet run aloud. Block Four is the untested part — the wrong answer has to be genuinely convincing or the vote is not close, and if the vote is not close the lesson does not land.