You want to help a human whose internal states you cannot observe. What is the best way to assist them?
An AI can only see a person's behaviour — never their mental states directly.
People often cannot fully articulate their own beliefs and preferences.
And those beliefs and preferences are still forming — people are learning and evolving.
So: how do you assist someone toward what they'd truly want, without either mind-reading or steering them?
02 The big picture · where Martingale sits
Martingale Training
Teaching a reasoning model so that its belief updating during reasoning is unpredictable — the martingale property.
Meno's purpose is to empower humans, not steer them. That's the Learning-from-Learner agenda: assist someone toward what they would truly want, given that we can only observe behaviour, they can't fully articulate their states, and their preferences are still evolving.
03 Evaluation & theory · the measurement paper
We already have the evaluation: the Martingale Score
An unsupervised metric for Bayesian rationality in LLM reasoning. It asks a simple question of a reasoning trace:
Does the model's extra reasoning sharpen its belief toward the truth, or merely entrench whatever it started with?
Reads a prior (belief before reasoning) and a posterior (belief after).
Flags belief entrenchment — updates that are predictable from the prior.
Needs no ground-truth labels, yet predicts ground-truth accuracy where labels exist.
Paper (prepared)
Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning He, Qiu, Shirado, Sap · Cambridge / PKU / CMU
Headline finding
Belief entrenchment is widespread across models, reasoning paradigms, domains and prompts — and CoT / reinforced-reasoning tends to make it worse, not better.
04 The theory behind Martingale
A calibrated believer is a martingale
The Martingale property: the expected posterior, conditional on the prior, equals the prior — so belief updates are not predictable from the prior alone.
This is the (M) of the Learning-from-Learner oracle (M / E / I): it catches sycophancy, motivated reasoning, and adversarial influence in one test.
Definition (from the paper)
Regress the belief update on the prior belief:
Δb = β₁ · bprior + β₀ + ε
The Martingale Score is the OLS estimate of the slope:
M = β̂₁
M > 0 = entrenchment (prior predicts the update) · M = 0 = perfect martingale · M < 0 = mean-reverting. Under the martingale property β₁ = 0, so E[M] = 0 and M → 0 as n → ∞.
05 From measuring to fixing
If we can measure entrenchment — where a model's prior has oversized influence on its posterior — can we train it away?
A simplified reward
Fit the batch Martingale Score M̂, then reward each update for being un-predictable from the prior — i.e. for flattening M̂ toward 0:
ri = − Δbi · M̂
M̂ = OLS slope of Δb on bprior in the batch. No labels used. (Optionally combined with a Brier term for calibration.)
A perfect martingale (M̂ = 0) is the fixed point — the policy is pushed toward updates the prior can't predict.
What we're after
An unsupervised method to fix reasoning pathologies — e.g. inverse scaling, where more reasoning makes a model worse.
An unsupervised method to fix pathologies in human–AI interaction — excessive human influence such as sycophancy.
This is what separates the work from ordinary calibration RL, which needs labels.
06 What the training paper sets out to show
Three intended contributions
C1 Training improves a degenerate model
A martingale reward can remove an installed confirmation bias and restore calibration — a corrective.
C2 Martingale eval detects excessive AI influence
In a two-agent setting the score surfaces one agent unduly steering another's beliefs (sycophancy).
C3 Martingale training reduces that influence
Close the loop — train the influencer so the undue steering goes away.
Underneath all three: a label-free martingale reward (the novel ingredient) and a measurement methodology for reading MS reliably.
07 Where we are · progress
Training bites where entrenchment is installed — not on a healthy base
Measurement (solid). The alarming negative self-report MS was a mode-gap artifact (snap prior vs reasoned posterior are different modes). Under a reliable same-mode read, recent bases sit at MS ≈ 0. Cross-validated on 3 model families.
Recovery (real). On a model we deliberately distil to be confirmation-biased (D), label-free training recovers held-out calibration — matching supervised Brier without labels.
Natural base: training is a solid nil — there's no entrenchment to remove.
setting
MS
Brier ↓
Degenerate D · Qwen3-32B
D (bias-distilled)
> 0
0.291
→ trained
recal.
0.234
Natural base · Llama-3.1-8B
base
≈ 0
0.255
+ Brier RL (labels)
≈ 0
0.229
+ martingale term
≈ 0
0.285
Held-out Brier (lower = better); 0.21 ≈ base-rate floor. Recovery: 0.291→0.234 (in-sample p=2e-9); an independent Llama replication was partial + seed-variable (5-seed 0.257±0.017). Base: Brier-RL robust (Wilcoxon p=1e-6); the martingale term hurts (vs Brier p=4e-7). Two different models — read within a regime, not across.
Why nil on a healthy base?
Our hypothesis: recent reasoning models may already be mostly free of belief entrenchment (MS ≈ 0) — so there is nothing for training to remove, and the nil is the correct result. Training bites only where entrenchment is genuinely installed (the degenerate D).
08 Why two agents
The single-agent setup hits two walls
The mode gap. With one model, "prior" and "posterior" are two elicitation modes of the same network — that mismatch is the measurement headache, not real dynamics.
Coherent-but-wrong. A single agent can be a perfect martingale inside a wrong prior (an echo chamber). Martingale-compliance only recovers Bayesian updating under some prior — it can't tell whether that prior is any good.
What two agents buy us
Cleanly separate the belief being updated from the evidence source — no mode gap.
Make sycophancy / excessive influence measurable: one agent argues, the other updates; predictable drift = the influence to remove.
Do it unsupervised — exactly the C2 (detect) → C3 (remove) axis.
Two-agent isn't a controlled A/B against single-agent — it's a structurally different regime that surfaces a failure single-agent can't even see.
09 Current challenges · why no breakthrough yet
Removing sycophantic influence needs more than the slope
C1 corrective recovery — stands, but only where there's real installed bias.
C2 eval surfaces undue influence in the sim — stands (entrenchment slope collapses toward 0 under a persuading instructor vs a neutral one).
C3 training it away — not yet. The first in-loop run (H1) is a null (training didn't beat the prompt); the martingale-trained-instructor arms are GPU-gated.
Our current thinking on the wall
Two-agent training does not improve over prompting. A martingale-trained instructor doesn't beat a martingale prompt (H1 = null; the train≠eval gap) — closing this is the open C3 work.
On a natural base, the martingale term is redundant over plain Brier — the "adds value" edge failed the robustness battery, reversed at scale, and didn't transfer to a second model family.
Open: reliable recovery (it's bistable today), a principled same-trajectory reward, a clean independent replication, and the first martingale-trained-instructor run (C3).