formalizing (II)
ghost gets handed everything it would get on a real morning. the full knowledge base. the live tape. every line of the prompt, tuned over a year. and then, instead of the one instruction it has seen ten thousand times — produce your thesis — it gets a different one.
grade your inputs. for every section i just gave you, tell me: did it change your answer, shade it, ride through to the output untouched, or do nothing at all.
here is why you have to ask it that way.
you cannot ask a model how it reached a conclusion. it will tell you — fluently, a clean chain of reasoning, delivered with total conviction — and the story will be a reconstruction, not a record. it narrates the process it should have followed, not the one it did. ask again and the story comes back slightly different. the model is an unreliable witness to its own mind, and the tell is that it never sounds unsure.
so you don’t ask it to remember. you catch it in the act. you hand it the exact production context — same data, same market, same instructions, down to the character — and swap only the final ask. not what would you do with this, but what of this did you actually use. now it isn’t recalling a process from memory. it’s reporting on the thing still in front of it.
what comes back is not flattering. one of its agents had been fed a whole interpretation matrix every morning — and had never once used it. noise, by the agent’s own grading. the strategist, made to account for itself, admitted it leaned on maybe a third of what it was given. so you cut what the gradings flagged. one prompt shed a hundred and seventy-eight lines. its output dropped from thirty-seven fields to twenty-four. the agent did not get dumber — it got honest. the framework is published in full.
and you do not take its word for it. the very same exercise that surfaces the dead weight also confabulates: one run invented an output field that was never in the schema — asserted with the exact conviction it used for everything true it said. that is the method restated: a model interrogating itself gives you leads, not verdicts. every cut gets checked against the real prompt and confirmed on a supervised run before it ships. the witness is useful precisely because you never trust it.
a model will always tell you a clean story about how it works. that story is the one thing you can’t use. interrogation is the other move: you stop listening to what it says about itself and watch what it does when you take a piece away. the witness is brilliant — and just as certain when it’s wrong. so you never take it at its word.