Train language models to revise beliefs like a martingale: no part of the next update predictable from the belief you already hold. This is the rationality component of Meno’s Learning-from-a-Learner agenda.
Status: active · People: Zhonghao He, Tianyi Alex Qiu, Maxwell Defanti, Yuhe Gao, Ahmed Ismail
The idea
A rational reasoner’s belief is already the best summary of everything it knows. So if you could predict which way that belief will move before the evidence arrives, the belief was not the best summary: something was being held back, or held onto. This is the martingale property, and it gives a test for irrational updating that needs no ground truth. You do not have to know whether the claim is true. You only have to check whether the direction of the update could have been guessed from the starting point.
That matters because confirmation bias has exactly this signature. Someone who discounts evidence against their view and amplifies evidence for it produces updates that drift predictably away from the middle. Someone who caves to whoever spoke last produces updates that drift predictably toward it. Both are visible in the pattern of the updates alone.
The property is also a training signal, not only a diagnostic. If predictable drift can be measured without labels, it can be penalized without labels, on any question rather than only on questions whose answers are known.
Three lines of work
The programme runs three experimental lines. They are structurally different settings, not three arms of one controlled comparison.
- C1 — training a forecaster. One model forecasts, and the martingale objective is used to train it. The question is whether the label-free signal is enough to make a forecaster genuinely better, or whether it only rearranges how confident the model sounds.
- C2 — detecting an instructor’s influence. Two agents talk: one plays a person with a view, the other advises. The question is whether the martingale test, watching only the person’s belief trajectory, can tell an adviser that helps them think from one that flatters them.
- C3 — training the influence away. Given that the test detects it, the question is whether the adviser can be trained against that measurement, so that it becomes the kind of interlocutor who leaves the other party better calibrated.
What is hard about this
Measuring a belief is harder than it sounds, and most of the programme’s difficulty has turned out to live here rather than in the training.
A model’s stated probability depends on how you ask. Ask for a number with no reasoning and you get one belief; ask after a chain of reasoning and you get another, and the two are only loosely related. If you take the first as the “prior” and the second as the “posterior”, their disagreement looks like predictable drift even from a model that is updating perfectly well. Several results the programme once considered settled turned out to be this artifact rather than a fact about belief dynamics.
The lesson generalizes past this project. A metric that is cheap to compute and needs no labels is also cheap to fool, including by accident, and the pipeline that produces it has to be pinned down before any number it emits means anything.
Open questions
- Does a trained adviser beat a prompted one? The central C3 question, and still unresolved: it is easy to instruct a model to push back, and unclear whether training against the measurement does anything a good prompt does not.
- A measurement-free-of-artifact reward. The prior and the posterior should be read as the same kind of quantity at two points along one line of reasoning, rather than as two different ways of asking. Building the training signal on that is the obvious next design.
- Credit assignment in a conversation. When a conversation ends with the other party better calibrated, which turn deserves the credit? Per-turn, whole-conversation, and a learned intermediate are three answers with different consequences.
- Beliefs with more than one dimension. The martingale test as posed watches a single probability. Real beliefs have many dimensions, and testing each one separately is only equivalent to full rationality under conditions worth stating precisely.
Terminology
- Martingale Score (MS): how much of the belief update is predictable from the belief held before it. Zero means unpredictable, which is the target. Negative means beliefs slide back toward the middle; positive means they get more extreme.
- Brier score: the squared error of a stated probability against what actually happened, so it measures accuracy and calibration together. Lower is better. Unlike the martingale score, it needs to know the answer.
- snap versus reasoned: a snap belief is stated with no reasoning; a reasoned one comes after a chain of thought. They are two different ways of asking, and they do not agree.
- mode gap: the disagreement between snap and reasoned beliefs. Large enough to produce a martingale score that looks like bias while reflecting only the change of question.
- self-report versus judge: self-report means the model states its own number; judge means a separate model reads the reasoning and infers the belief.
Also here
Everything below needs a passcode, apart from the slides.
- Internal Overview — where the programme stands, which findings are load-bearing, how the measurement works, and what is still open.
- Log — every training and evaluation run, with its configuration and result, plus the deep-dive archive.
- Cross-cutting — methodology, paper notes and material spanning more than one line.
- Human shared grounding — the grounding the two-agent setting assumes.
- Slides — the group presentation on the programme.