Hallucination

An adviser is reading the AI's notes from Alan Pritchard's annual review before approving them. The meeting ran short and was mostly about his daughter's house deposit. Near the end, the notes say "Alan confirmed he remains comfortable with the risk level of his portfolio and understands its value can fall." The adviser plays back the recording. Nobody mentioned risk. Approved as they stand, the notes would put a risk conversation on file that never happened.

The AI writes by choosing words that are likely to come next, given the conversation and what it learned in training. In annual review notes, a line confirming attitude to risk is a likely one. It can write that line whether or not anyone said it. The way AI models are trained and tested rewards guessing over admitting uncertainty, so nothing flags the line as a guess.

That is a hallucination. A hallucination is a statement an AI makes up and presents as fact. It is not in the material the AI was given, and it is not true. The everyday word suggests a glitch, as if the AI saw something that isn't there. The invented line comes from the same process as the accurate ones, working as it always does.

The file check is the firm's control here, and it is good at finding gaps, such as an annual review with no risk discussion on file. This hallucination fills that gap with the very sentence the check looks for. The file would pass. The notes show what the AI wrote, and only the recording shows what was said. A harness that links each line of the notes to its place in the recording makes checking them before approval quicker. Without one, each line has to be found in the recording by hand.