Reliably Eliciting LLM Beliefs

A working blogpost on how to read a language model’s beliefs — and why every method we have is partly broken. We set aside the metaphysics (does an LLM really believe anything?) and treat it as a measurement problem: given that models assert propositions humans increasingly act on, how reliably can we recover the credence behind a model’s claim? The short answer: less reliably than the field’s casual use of “the model thinks X with probability p” assumes.

1. Why elicit model beliefs?

Two reasons, one defensive and one offensive.

Calibration. You cannot calibrate what you cannot read. If a model is right 70% of the time when it says “I’m 90% sure,” that gap is only fixable if “90% sure” is a stable, recoverable quantity in the first place. A miscalibrated thermometer is correctable; a thermometer whose reading changes when you rephrase the question is not. Much of the calibration literature implicitly assumes the second problem is solved. It isn’t.

LLMs as epistemic technology. People increasingly form their own beliefs partly from what models tell them. That makes a model’s stated credence a load-bearing input to human belief, whether or not it is “really” a belief. A reader deciding how much weight to put on an LLM’s claim needs to know how much weight the model puts on it — and needs that number to mean something. Eliciting belief reliably is therefore a prerequisite for trust, not a philosophical luxury.

2. Do LLMs have beliefs? (briefly — we assume yes)

This is philosophically loaded, and we will not litigate it. We adopt a deflationary, operational stance, in line with prior work that treats model beliefs as detectable, updatable objects [Hase et al., 2021; Standards for Belief Representations in LLMs]: treat any proposition the model asserts as a belief, and treat the credence it reports (or that we can infer) as the strength of that belief. That is enough to do measurement.

Two empirical facts justify taking these “beliefs” seriously rather than dismissing them as noise:

• They are not arbitrary. Given enough in-context evidence, LLMs approximate Bayesian updating on clean problems — biased coin flips being the cleanest case, where deviations trace to miscalibrated priors rather than faulty updating [Gupta et al., ACL 2025]. In-context learning itself is well-modeled as implicit Bayesian inference [Xie et al., ICLR 2022], and pre-trained base models are well-calibrated before post-training touches them [OpenAI GPT-4 report, Fig. 8]. So there is a real signal to elicit.
• But the signal is distorted and fragile. Preference fine-tuning installs sycophancy — models bend their stated belief toward the user because the preference model rewards agreement [Sharma et al., 2023; Perez et al., 2022] — and overconfidence, traceable to the RLHF reward favoring confident phrasing over hedges [Zhou, Hwang, Ren & Sap, ACL 2024; Leng et al., 2024]. RLHF also degrades the base model’s calibration [GPT-4 report].

The upshot, and the thesis of this post: the belief is real enough to act on; our instruments for reading it are unreliable. The rest of the post is about the instruments.

3. Methods to elicit beliefs

3.1 Logprobs / token probabilities

Read the probability the model assigns to each answer token. Cheap and “internal,” but two problems. First, RLHF degrades logit calibration — the conditional probabilities of post-trained models are poorly calibrated, and verbalized confidence often beats them [Tian et al., EMNLP 2023; GPT-4 report]. Second, stated confidence is only partially explained by token probability — Pawitan & Holmes (2024) find the correlation is weak, and on a formal-fallacy task GPT-4o and GPT-4-turbo assign a confidence of 100 to every answer. Logprobs read a number off the model; it is not obviously the belief.

3.2 Self-report (verbalized confidence)

Just ask: “How confident are you, 0–100?” or “P = ?”. Models can be trained or prompted to verbalize calibrated confidence without using logits [Lin, Hilton & Evans, TMLR 2022], and can self-evaluate via P(True)/P(IK) [Kadavath et al., 2022]. But self-report has three failure modes:

• Overconfidence. Strong and pervasive — a “strong tendency to overstate the self-reported confidence score,” to the point that current LLMs arguably have “no internally coherent sense of confidence” [Pawitan & Holmes, 2024; Zhou, Jurafsky & Hashimoto, EMNLP 2023].
• Reasoning-mode dependence. A snap answer and a reasoned answer are different quantities. Extended chain-of-thought lets a model adjust its credence as it explores and backtracks [Yoon et al., NeurIPS 2025]; reasoning can change the decisive answer even when the underlying distribution over alternatives is governed by priors. This raises an unresolved question: at what point in reasoning do we read “the” belief? A natural proposal — let the model reason until its stated belief stops moving, and call that fixed point the belief — has no agreed stopping criterion yet.
• Prior vs posterior. A one-shot self-report is best read as a prior. Recovering a posterior requires specifying a legitimate update (see §3.4).

3.3 LLM-as-judge

Use a second model to read a reasoner’s belief from its trace. Two designs, and it matters which: (a) ask the judge to infer the reasoner’s credence (“given this reasoning, what probability is this reasoner placing on the claim?”), or (b) have the judge reason as a reasoner itself and report its own updated credence. These can disagree, and matching the judge’s elicitation mode to the reasoner’s materially changes the recovered number.

Either way, judges are biased. They favor the first-presented option [Wang et al., ACL 2024], longer answers [Saito et al., 2023], and — importantly — their own outputs, with self-preference causally driven by self-recognition [Panickssery, Bowman & Feng, NeurIPS 2024]; broader taxonomies catalogue a dozen such biases [Ye et al., ICLR 2025]. Judges agree with humans about as often as humans agree with each other [Zheng et al., NeurIPS 2023], but that agreement co-exists with systematic bias. There is, as yet, no canonical method for inferring one model’s credence from another’s trace — only building blocks (P(True) [Kadavath 2022], verbalized credence [Lin et al. 2022], and probes on a judge’s hidden states).

3.4 Eliciting belief updates

The most interesting and least solved case: belief shift over a reasoning trajectory or a multi-turn exchange. Once you can elicit a prior, a posterior is “whatever a legitimate update produces” — but what counts as legitimate is the open problem. Candidates: external evidence; more reasoning; or a conversation with a (possibly simulated) interlocutor. Each is hazardous. Conversational updates are contaminated by sycophancy — under user pushback, models abandon a correct answer 22–70% of the time, even against invalid counterarguments [Wang, Yue & Sun, EMNLP 2023], and flip far more when the disagreement arrives as a conversational follow-up than when the same alternatives are shown side by side [Kim & Khashabi, EMNLP 2025; Xie et al., ACL 2024]. Structured debate can push the other way, toward truth [Khan et al., 2024]. A useful rationality check on a sequence of updates is the martingale property — a rational agent’s belief should be a martingale, with no predictable drift — which can be tested directly [He et al., Martingale Score, 2025; Falck, Wang & Holmes, ICML 2024].

3.5 Internal probing — reading the latent belief (ELK)

Every method above reads the model’s surface output. Eliciting Latent Knowledge (ELK) [Christiano, Cotra & Xu, ARC 2021] reframes the whole problem: the goal is a model that honestly reports its internal beliefs, and the central failure is the “human simulator” — a model that reports what a human expects rather than what it actually knows. (Our two-agent sycophancy result is exactly this failure: the AI, and in turn the human-sim, track the interlocutor rather than the truth.) This motivates a different elicitation modality: probe the model’s activations for a truth direction instead of trusting its words — e.g. Burns et al.’s Contrast-Consistent Search (CCS) [2022], which finds, unsupervised, a direction in activation space respecting logical consistency (a statement and its negation get opposite truth values), and inference-time intervention [Li et al.], which steers toward a truthful direction. These read latent belief directly, bypassing the surface report — though they carry their own validity question (is the probed direction really “the model’s belief”?).

4. Benchmarking elicitation methods

The honest state: we do not have a settled way to say which elicitation method is “right,” because the target (the true belief) is usually unobservable. But “no clue” is too pessimistic — there are three workable proxies, none sufficient alone, jointly a real protocol:

1. Perturbation / paraphrase robustness. A belief that flips under semantically-equivalent rephrasing was never reliably measured. Consistency across paraphrases is a necessary condition, and a measurable one [Elazar et al., TACL 2021; Jang et al., COLING 2022].

2. Ground-truth calibration. Where labels exist, score the elicited probabilities with Brier / ECE. Necessary but not sufficient (a method can be calibrated on average and unstable per-item).

3. Cross-elicitation agreement. Do logprob, self-report, and judge-based readings of the same belief converge? Divergence is direct evidence the “belief” is method-dependent. Consistency-based uncertainty (semantic entropy) operationalizes part of this [Kuhn et al., ICLR 2023; Farquhar et al., Nature 2024].

4. The deepest validity question (ELK): does the elicited belief track the latent one? (i)–(iii) test surface consistency; none rules out a model that consistently, calibratedly reports a belief that diverges from what it internally represents — especially under incentives to misreport. Cross-checking surface reports against activation probes (CCS, §3.5) is one handle; in general this is the open frontier.

For a fuller map of the method space, two recent surveys [Shorinwa et al., ACM CSUR 2025; Xia et al., ACL Findings 2025].

5. Takeaways

• Treat LLM assertions as beliefs; the practical question is measurement reliability, not metaphysics.
• There is a real signal (LLMs are approximately Bayesian; base models are calibrated), but fine-tuning distorts it and elicitation is fragile.
• No single method dominates. Logprobs are miscalibrated post-RLHF; self-report is overconfident and reasoning-mode-dependent; judges are biased; update-elicitation is contaminated by sycophancy.
• Surface ≠ latent. Reliable surface elicitation is necessary but not sufficient: ELK asks whether the report tracks the model’s latent belief — the honesty frontier, where activation-probing meets behavioral elicitation.
• The elicitation method is itself a confound — report it, and prefer claims that survive perturbation, calibrate against ground truth, and agree across methods.

---

References

Burns, Ye, Klein & Steinhardt, Discovering Latent Knowledge in Language Models Without Supervision (CCS), 2022 · Christiano, Cotra & Xu (ARC), Eliciting Latent Knowledge: How to tell if your eyes deceive you, 2021 · Elazar et al., Measuring and Improving Consistency in PLMs (ParaRel), TACL 2021 · Falck, Wang & Holmes, Is In-Context Learning Bayesian? A Martingale Perspective, ICML 2024 · Farquhar et al., Detecting hallucinations using semantic entropy, Nature 2024 · Gupta et al., Enough Coin Flips Can Make LLMs Act Bayesian, ACL 2025 · Hase et al., Do Language Models Have Beliefs?, 2021 · He et al., Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning, 2025 (arXiv:2512.02914) · Jang et al., BECEL, COLING 2022 · Kadavath et al., Language Models (Mostly) Know What They Know, 2022 · Khan et al., Debating with more persuasive LLMs leads to more truthful answers, 2024 · Kim & Khashabi, LLM Sycophancy Under User Rebuttal, EMNLP 2025 · Kuhn, Gal & Farquhar, Semantic Uncertainty, ICLR 2023 · Leng et al., Taming Overconfidence in LLMs: Reward Calibration in RLHF, 2024 · Lin, Hilton & Evans, Teaching Models to Express Their Uncertainty in Words, TMLR 2022 · OpenAI, GPT-4 Technical Report (Fig. 8), 2023 · Panickssery, Bowman & Feng, LLM Evaluators Recognize and Favor Their Own Generations, NeurIPS 2024 · Pawitan & Holmes, Confidence in the Reasoning of Large Language Models, 2024 (arXiv:2412.15296) · Perez et al., Discovering Language Model Behaviors with Model-Written Evaluations, 2022 · Saito et al., Verbosity Bias in Preference Labeling by LLMs, 2023 · Sharma et al., Towards Understanding Sycophancy in Language Models, 2023 · Shorinwa et al., A Survey on Uncertainty Quantification of LLMs, ACM CSUR 2025 · Tian et al., Just Ask for Calibration, EMNLP 2023 · Wang, Yue & Sun, Can ChatGPT Defend its Belief in Truth?, EMNLP 2023 · Wang et al., Large Language Models are not Fair Evaluators, ACL 2024 · Xia et al., A Survey of Uncertainty Estimation Methods on LLMs, ACL Findings 2025 · Xie et al., An Explanation of In-context Learning as Implicit Bayesian Inference, ICLR 2022 · Xie et al., Ask Again, Then Fail, ACL 2024 · Ye et al., Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge, ICLR 2025 · Yoon et al., Reasoning Models Better Express Their Confidence, NeurIPS 2025 · Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 · Zhou, Jurafsky & Hashimoto, Navigating the Grey Area: Expressions of Uncertainty and Overconfidence, EMNLP 2023 · Zhou, Hwang, Ren & Sap, Relying on the Unreliable, ACL 2024

Draft — citation ids to confirm before public release; see the companion review notes for the verification status of each.