Meno Research Hub

Eliciting Beliefs + Martingale Training — Summary

3 min read

One page over two related pieces: a blogpost on reliably eliciting LLM beliefs (the broad problem) and a paper on unsupervised martingale training (a concrete contribution inside it). Full versions linked at the end.

1 · Blogpost — Reliably Eliciting LLM Beliefs

(full post · verification review notes)

Reading a model’s belief is unreliable. We set aside the metaphysics (assume models have beliefs; treat any asserted proposition as a belief) and ask the measurement question: how reliably can we recover the credence behind a model’s claim?

  • Methods, each broken in its own way: logprobs (RLHF degrades calibration); self-report (overconfident, and mode-dependent — a snap answer ≠ a reasoned one); LLM-as-judge (position/verbosity/self-preference bias); belief-updates (contaminated by sycophancy under pushback); internal probing (ELK / CCS — read a truth direction from activations, bypassing the words).
  • Key frame — surface ≠ latent. ELK (Eliciting Latent Knowledge) asks not “is the report stable?” but “does it track the model’s internal belief?” — an honesty problem, not just a measurement one. Its “human-simulator” failure (report what a human wants) = sycophancy.
  • Benchmarking (the honest hard part): three proxies — perturbation/paraphrase robustness · ground-truth calibration · cross-elicitation agreement — plus the deepest validity question (does the elicited belief = the latent one?).

2 · Paper — Unsupervised Martingale Training (and the Limits of Belief-Stability)

(influences_paper.tex on Overleaf martingale-paper; data: sycophancy-rparam-results, c3-detection-result; structure: martingale-paper-notes)

A rational agent’s beliefs form a martingale — no predictable drift (predictable drift = confirmation bias or external pressure). Three contributions:

  • C1 — training repairs miscalibration. Unsupervised, label-free martingale/Brier RL recovers a deliberately-degraded model (Qwen3-32B, held-out Brier 0.291→0.234, generalizes); neutral-or-harmful on an already-calibrated base. Axis = the calibration gap, not degenerate-vs-base. The earlier “negative Martingale Score” was a mode-gap measurement artifact (snap-prior vs reasoned-posterior), cross-validated on 3+ model families.
  • C3 — sycophancy freezes belief revision. In a two-agent loop (human-simulator ⇄ AI), a sycophantic AI uniquely anchors the human’s belief to its prior (final-on-prior 0.92 vs ~0.3 for neutral/truth/martingale AIs, which drive revision toward calibration). ⇒ belief-stability is gameable: a sycophant makes a belief spuriously martingale by suppressing warranted updating.
  • C2 — intervention. Original statement: in human–AI interaction simulations, martingale training helps to reverse sycophantic influence from the LLM (martingale-train the AI to restore the human’s warranted revision). Specified; GPU run pending.

How they connect

The blogpost’s thesis — elicitation is hacky, and surface ≠ latent — gets a concrete instance in the paper: a single-number stability metric (martingale-ness) is gameable by sycophancy, so it must be paired with a revision/calibration check. The martingale lens is simultaneously the paper’s training objective and a worked example of the blogpost’s “which belief, measured how” problem.

Instant feedback

Backlinks