Primer answering Zhonghao’s foundations questions (2026-06-06), from the belief/preference meeting + the learning-from-learner writeup. Two parts: Part 1 explains the concepts you said were unclear; Part 2 engages your own reasoning on the ones you took a position on — marking where it’s solid and where the load-bearing gap is. Companion to lfl-theory.
Part 1 — concept explainers
What is a Markov reward? (and “Σγᵗrₜ over states”, and “no Markov reward can represent”)
A Markov state is a description of the situation that contains everything you need to predict the future — the past adds nothing once you know it. (“Markov” = memoryless: the future depends on the present state, not the path that led there.)
A Markov reward assigns a scalar r(s) (or r(s,a)) to each state, and the value of a whole trajectory is the discounted sum:
R(τ) = Σ_t γᵗ · r(s_t) — i.e. r(s₀) + γ·r(s₁) + γ²·r(s₂) + …
γ ∈ (0,1) is the discount (future reward worth less). The key structural commitment: trajectory value is additive over per-state rewards. You score each state independently and add them up.
“No Markov reward can represent X” means: there is an ordering you want over trajectories/policies (you can say “A is better than B”) for which no choice of per-state r(s) reproduces that ordering via the discounted sum. Abel et al. (2021, On the Expressivity of Markov Reward) proved such orderings exist. Concretely, additivity can’t express:
- “do A, but only if you haven’t already done it twice” — value depends on history, not the current state.
- process preferences: “reach the goal fairly” / “without lying on the way” — two trajectories with identical end-states but different paths must score differently, which a sum-over-states can’t do (the states are the same).
- non-additive trade-offs: “I want some of X and some of Y, but not all-X” — additivity always says more-of-the-best-state is better.
So “no Markov reward represents your preference” = you can rank the options, but that ranking is not a discounted sum of state-scores → the reward formalism is too weak to hold your actual preference. This is one of the concrete ways “preference = reward” leaks (see Part 2).
What is an attractor?
An attractor is a state (or set of states) a dynamical system converges to and stays near — once you’re in its basin of attraction (the set of starting points that flow toward it), the dynamics pull you in. Your “local optimum” picture is right for the simplest kind (a fixed-point attractor = a valley a ball rolls into), but the concept is more general:
- fixed point — converges to one state (a settled opinion).
- limit cycle — converges to a repeating loop (oscillating between two views forever).
- strange attractor — converges to a bounded but never-repeating region (chaotic-but-confined).
For belief dynamics the useful mental image: the space of possible belief-states is a landscape; the update process is the flow; attractors are where beliefs settle. An echo chamber is a low-value attractor — a stable belief configuration with a wide basin, self-reinforcing so perturbations decay back toward it. The important property is not “it’s an optimum” but it’s stable and self-restoring: that’s exactly why simply showing one counter-fact often fails (the basin pulls back). More below.
Metacognition — what it is and why it matters (and “temporal autoregression”)
Metacognition = cognition about cognition — modeling a mind (your own or another’s) as a thing with beliefs that update, rather than predicting its outputs at face value. The distinction that matters here:
- Object-level prediction: “what will this person say/do next?” — predict the text/action.
- Metacognitive prediction: “what does this person believe, how will that belief change as evidence arrives, and why?” — predict the latent state and its update rule.
Why current LLMs lack it: standard autoregression trains a model to predict the next token of text at face value. Nothing in that objective rewards building a model of the author’s epistemic state — it’s rewarded for surface continuation, not for “what would this agent come to believe.” So it learns to mimic what people say, not to model how people think/update.
Temporal autoregression (Tianyi’s proposed fix): instead of predicting the next token, train the model to predict the future actions and beliefs of an agent over time — given an agent’s private observations + every other agent’s belief-changes, predict their next belief/action. The claim: this objective forces a metacognitive model, because you cannot predict how someone’s belief will move many steps out without internally representing their belief-state and update rule. (Predicting long-horizon updates is unachievable by surface heuristics → the model is pressured to build the real thing.) This is the bridge to “predicting future human updates” in Part 2 — the training signal that would make an AI good at it.
Outcomes razor & hypothesis selection
Start with the ordinary Occam’s razor: prefer the simpler hypothesis. The problem: “simpler” is slippery, and a naive likelihood-maximizer just picks whatever single hypothesis best fits the data, which overfits.
The outcomes razor (from Singular Learning Theory, SLT) is a principled version. The idea: don’t ask “which single hypothesis has highest likelihood?” — ask “which hypotheses occupy a broad region of parameter space?” A hypothesis that’s correct across a wide, flat basin of parameter settings is preferred over one that’s correct only at a sharp, isolated point — even if the sharp point fits slightly better. The broad region is more robust and more probable under the posterior (it has more “mass”).
The clean analogy is BIC (Bayesian Information Criterion): when you integrate the likelihood over a parameter region (the Bayesian thing to do), a broad-but-slightly-lower peak can beat a tall-but-narrow spike, because the integral (the evidence) weighs volume × height, not just height. SLT generalizes this to singular models (neural nets), where the geometry is degenerate (many parameter settings give the same function) — the right complexity measure is the Real Log Canonical Threshold (RLCT), which counts the effective dimensionality of a solution basin.
Why it matters / “hypothesis selection”: when an AI infers a human’s latent reward/belief from behavior, many hypotheses fit (the identifiability problem). The outcomes razor is a principled tie-breaker: prefer the explanation occupying the broadest basin — the “ridge” over the isolated “peak.” Tianyi’s twist: deep nets trained by SGD may already implement the outcomes razor (SGD is biased toward flat/broad basins) — so it might not be a missing ingredient that needs explicit engineering, unlike the temporal-autoregression signal, which is genuinely absent from the training objective. That’s the “(1) more certain issue [missing TA signal] vs (2) less certain, maybe-architecture-resolved [missing razor]” split in the meeting notes.
One-line map of the two ideas together: temporal autoregression is the missing training signal that would make a model metacognitive; the outcomes razor is the missing inductive bias for choosing among the many latent-models that fit — and SGD may supply the second for free, which is why the first is the higher-priority bottleneck.
Part 2 — engaging your reasoning
For each, I mark [your position] → [where it’s solid] → [the gap / crux].
vNM: “if the theorem is that bad, why is RLHF still built on it?”
Your position: humans violate the vNM axioms (incomparability, cycles, Allais), so the preference≈utility theorem doesn’t apply — yet RLHF still assumes it. Contradiction?
Where it’s solid: the violations are real and the theorem genuinely doesn’t strictly hold for humans.
The gap: “the theorem fails” ≠ “the method is useless.” RLHF uses vNM as a local approximation that you can optimize, not as a literal truth about humans. Three reasons it persists despite the violations: (1) you can only optimize a scalar — a raw preference relation isn’t differentiable; the Bradley-Terry reward model exists precisely to turn comparisons into a number to ascend. (2) Near the data it’s fine — locally, on the distribution you trained on, a fitted scalar tracks behavior well; the violations bite at the edges (extreme lotteries, intransitive cycles), not in the middle. (3) No deployed alternative — until recently nobody had a trainable formalism for non-vNM preferences. The crux you’re circling: RLHF isn’t wrong because the axioms fail; it’s lossy — it projects a structured, context-dependent, possibly-incomplete preference onto a single scalar, and then people reify the projection (treat the fitted reward as the person’s actual values). The error is the reification, not the use. This is exactly why your writeup drops s_obs (observed/revealed preference) from the state space — you refuse to identify the scalar-you-can-measure with the preference-you-care-about. That’s the same instinct, formalized.
”Reward = preference” and revealed ≠ true latent preference
Your position (restated): rewards go to whatever scores higher; vNM guarantees that’s higher-utility; but human actions (revealed preference) don’t represent true latent preference — they’re chosen under influence/manipulation/missing-info/irrationality — and you can’t distinguish “I acted on my true preference” from “I acted under distortion,” yet the action is taken as the preference anyway.
This is correct and it is the heart of the matter. It’s the classic objection (cf. the radical-interpretation problem — Lewis 1974 — and the irrationality-degeneracy of inverse RL: you can’t separate “what they want” from “how they err” from behavior alone). Your move in the writeup is the right one: stop treating revealed preference as the target. Replace it with ideal preference (what they’d endorse with the missing information + corrected beliefs), and make the human’s belief-state an explicit latent variable so that “acted under false belief” and “acted on true preference” become different states, not indistinguishable behavior. The remaining crux: this only works if you can actually correct the beliefs (your hyp:belief-gap) and elicit the preference cleanly once beliefs are fixed — and the elicitation itself (“which do you prefer, A or B?”) makes the human form beliefs about what A and B mean, so belief and preference re-entangle at the query. You concede this (Remark on entanglement); the open question is whether the entanglement is benign (a measurement-noise issue) or load-bearing (it reintroduces exactly the confound you removed).
Path dependence — “why is it an issue? different attractors seem fine”
Your position: if humans land on different attractors, so what? The worry is only a low-value attractor (echo chamber), and that’s solvable by belief/preference separation — treat the echo chamber as a belief problem, correct the mistaken beliefs as you go, and a human whose beliefs match reality can climb out of the low-value attractor.
Where it’s solid: you’ve correctly identified that path-dependence is only alarming when paths lead to differently-valued destinations, and that belief-correction is the lever. That’s the right diagnosis of the mechanism.
The gap / the crux — this is the subtle one. Path dependence isn’t a problem because “different attractors exist”; it’s a problem because it makes “the ideal preference” not well-defined. Your whole framework targets the ideal preference s_pref* — a single thing the AI should converge to. But if the human’s belief/preference evolution is path-dependent, then there is no single s_pref* — there’s s_pref*(path₁), s_pref*(path₂), … and the AI’s own interventions choose the path. So the target the AI is optimizing toward is partly created by the AI’s choices, not just discovered. That’s the deep issue Pierce flagged (“long-horizon RL converges to the wrong attractor if the update process is biased”) and why he puts path-dependence before horizon: a longer optimization horizon doesn’t help if the destination itself depends on how you got there — you’ll just confidently converge to whichever attractor your own nudges induced. Your belief-correction answer is partly right but assumes belief-correction is itself path-independent — that “show the facts, beliefs converge to truth” reaches the same place regardless of order/framing. Empirically that’s often false (motivated reasoning, order effects, identity-protective cognition) — which is exactly the echo-chamber case, where the basin pulls back. So: if hyp:belief-gap holds and belief-correction is order-independent, your answer works and path-dependence dissolves. If not, path-dependence is real and “track s_pref*” silently weakens to “track whichever destination this trajectory happened to reach.” The crux is whether belief-correction is path-independent — and your framework currently assumes it without flagging the assumption.
Active identification — literature, and does it really solve unidentifiability?
Your position: love the framing — the assistant actively tests hypotheses about the human’s latent state rather than passively waiting; you were already keen on the interventionist version (make humans more Bayesian, present world-facts to confront beliefs) but hadn’t realized it’s called “active identification” or that it addresses reward-unidentifiability.
Good news: it’s real, it has a literature, AND your writeup already does it. Not a made-up term — it’s the intersection of optimal experimental design / active learning (choose the query that maximizes expected information gain), active inverse RL / preference elicitation (Sadigh, Biyik et al. — pick the comparison query that most disambiguates the reward), and the active-learning-solves-identifiability result (a passive observer can’t separate many rewards behind one behavior, but an active querant can, because it can pose the question whose answer differs across the competing hypotheses). In your writeup this is condition I3 of your Identifiability proposition — “comprehensive / Fisher-information-positive-definite active query selection” — implemented as the information-gain term in Algorithm 1. So you formally get TV-convergence to a point mass on s_pref*. You’d already built the thing you were excited about.
The gap / crux — and it’s a sharp one, from your own results. Active identification solves unidentifiability only for the preference the human acts on. But your Impossibility-under-Persistent-Akrasia proposition shows active querying is powerless exactly when behavior is governed by θ̂ ≠ θ* — when the human reliably acts against their endorsed preference (weakness of will, addiction, higher-order desire). No query disambiguates “wants X but akratically chooses Y” from “wants Y,” because both produce choosing Y. So: active identification identifies the acted-upon preference, not necessarily the endorsed/ideal one. The unidentifiability it kills is “many rewards, same behavior when behavior tracks the reward.” The unidentifiability it can’t kill is “behavior systematically mistracks the reward” — which is the hard case and the one ideal preference is supposed to be about. The missing piece is a behavior-bypassing channel (reflective endorsement, meta-preference queries, “do you want to want this?”) that doesn’t route through the akratic action. That’s the live research gap your framework points at but hasn’t filled.
”Why predict future human belief-updates? — accurate prediction ⇒ effective intervention. True?”
Your intuition is right but slightly under-specified. Yes: if you can predict how a human’s mental state moves, you can choose interventions that steer it — prediction is the model you plan against. Two refinements that matter for your project: (1) the reason it’s necessary (not just nice) is anti-entrenchment — an AI that optimizes the human’s current preference freezes them (RLHF entrenchment); to target the ideal preference you must model what they’d believe/endorse after the updates they haven’t made yet → you need the forward model P (your “predictive system”). Prediction is what makes the target ideal rather than current. (2) The danger twin: prediction-enables-intervention cuts both ways — the same forward model that lets you help someone reach their reflective endorsement lets you manipulate them toward a destination you chose. What separates the two is precisely the path-dependence + belief/preference machinery above: helping = correcting beliefs toward an external truth-standard and eliciting (not choosing) preferences; manipulating = steering preference-formation toward your target. So “accurate prediction ⇒ effective intervention” is true, and that’s exactly why the belief/preference boundary is the entire safety surface — it’s the line between teaching and steering.
Echo chamber & interventionism — your reply to Tianyi
Your position: (a) it’s belief-state not preference-state (echo chambers are almost entirely about beliefs); (b) so intervening is fine — we should be interventionist on beliefs (mistaken/biased beliefs have a ground truth, correcting them is generally good); non-intervention is right only for preferences (paternalist, no ground truth).
Where it’s solid — and you’re right against the strong form of Tianyi’s pushback. The is/ought (direction-of-fit) split genuinely defuses the “intervening = paternalism” worry: correcting a belief defers to an external standard (the world), not the AI’s values, so it isn’t paternalism. And much echo-chamber damage is belief-mediated and self-corrects once the false factual beliefs are fixed. Your “teacher about facts, servant about values” is a clean principle.
The two gaps to hold. (1) Echo chambers also shape terminal values, not only beliefs — purity, loyalty, in-group identity, a taste for outrage — with no false factual belief in the chain to correct. For that residue Tianyi’s point survives: flagging it requires evaluating preference formation, which your non-intervention-on-preferences rule forbids. The principled patch (and it bridges you both) is a procedural/contingency test: a value is “captured” iff it’s path-dependent on the manipulation — you’d hold a different value had the echo chamber not intervened. That’s a counterfactual on preference-formation, not a substitution of the AI’s values, so it respects your anti-paternalism rule while reaching the part beliefs can’t. (Note this is the same path-dependence machinery again — the project keeps rediscovering it.) (2) “It’s a belief, so correcting it is good” is only as safe as the is/belief-vs-preference classification. Anything the AI wants to push that it can frame as a factual belief escapes your anti-paternalism guard — and value-laden “beliefs” (e.g. “this kind of life is more meaningful”) are exactly where teacher-mode quietly becomes preference-paternalism. Your framework’s safety therefore rests entirely on a clean is/belief/preference typing of each claim — which your own entanglement remark admits is an idealization. So your interventionist instinct is correct and its whole safety margin lives in one classification step that isn’t yet operationalized. That’s the crux worth your attention.
The thread running through all of it
Three of your answers (path-dependence, predicting-updates, echo-chamber) bottomed out at the same place: is the human’s belief/preference formation path-dependent, and if so who owns the path? And two more (reward=preference, interventionism) bottomed out at: is the is/belief/preference boundary clean enough to be load-bearing? Those are your two real cruxes. The good theoretical instinct you keep showing is typing things as different objects to dissolve a confound (dropping s_obs; is/ought; belief-vs-preference). The recurring blind spot is assuming the dissolving move is clean/path-independent without flagging it — the entanglement and the path-dependence are where the assumed-away difficulty actually lives. Noticing that pattern is, I think, the single most useful thing for building the intuition you said you wanted.
Part 3 — the LFL decomposition, with references
Added 2026-06-06, written up from the channel thread. Faithful to Tianyi’s writeup (
tianyi-may-lfl.md); every claim is tied to established literature, Tianyi’s doc (§/Remark/Conjecture/Open-Question), or Zhonghao’s paper (Proposition). Flags proven vs. open throughout.
A. The decomposition tree
Tianyi (Decomposition 2.3) factors the whole problem along the three orthogonal obstacles to a belief being truth-tracking (§2.1), each with a matched cure (§2.2):
LFL = Coherence (incoherence → Coherence Optimization)
+ Reflective Equilibrium(reflective instability → RE-based reward)
+ Bayesian Rationality (path dependence → §3, §4)
Bayesian Rationality = Martingale + Temporal Autoregression + Metacognition
Metacognition = Metacognitive Information (§3, regular) + MOR (§4, singular)
There is a dependency order (Rec 5.1): path-dependence is handled first, because RE is only well-defined as an optimization target once beliefs cannot converge to multiple attractors.
B. Bayesian Rationality vs. the M/E/I oracle
Zhonghao’s oracle tests a single agent’s update trajectory with Martingale + Exchangeability + Informativeness; Tianyi’s branch constructs Bayesian rationality with Martingale + Temporal Autoregression + Metacognition. They answer different questions — certify vs. construct — and map onto each other:
| Zhonghao’s oracle (test) | Tianyi’s branch (construct) | relation |
|---|---|---|
| Martingale | Martingale | identical — the shared anchor |
| Informativeness | Temporal Autoregression | both rule out the vacuous constant-predictor (trivially martingale, learns nothing); I is the test, TA the constructive target |
| Exchangeability (order-invariance) | — (it is the goal) | E = a test for path-independence; path-dependence is failure-mode #3, so the whole branch exists to achieve what E tests |
| — | Metacognition | the genuinely new layer: multi-agent theory-of-mind |
Is M/E/I sufficient? As a single-agent calibration test it is strong and necessary, but it cannot catch the echo-chamber/lock-in failure — an agent can be perfectly calibrated within a wrong source-reliability prior (Tianyi §3.2: martingale-compliance only recovers Bayesian inference under some prior+likelihood, which can be the bad one). That gap is exactly what Metacognition fills.
C. Why the temporal axis — the process lives on it
Token-autoregression predicts surface text from co-occurrence; it has no reason to represent a hidden belief or how it moves. Temporal autoregression predicts future beliefs/actions, which forces a latent model (the non-cheatable target). Tianyi:
Remark 3.12. “The operative sample space is the universal Ω, whose coordinates include each βⱼ, so ‘predict peers’ belief change’ is itself metacognitive inference.” Recommendation 5.2. Train with the per-step target
(its private observation + every other agent's belief change), using the realized ω⁽ᵗ⁾ trajectory as supervision.
Rigorous handle (Conjecture 3.11): under pooled regularity (Assumption 3.10), optimal predictive accuracy on that temporal stream ⇒ effectively learning every agent’s private observations ⇒ beliefs converge a.s. to the joint Bayesian posterior. The temporal axis is where the update process (the real target) lives; a static/marginal model is a snapshot with no information about the update.
Object-level updates move mass on the proposition; metacognitive updates move mass between “signal correlated with truth” (C) and “uncorrelated” (UC) — the war/UFO walk-through, Example 3.2. The order of observations drives which update fires. Lit on sequential-prediction ⇒ latent world-model / theory-of-mind: predictive coding (Rao & Ballard 1999); World Models (Ha & Schmidhuber 2018); Machine Theory of Mind (Rabinowitz et al. 2018).
D. MOR and Singular Learning Theory
When many models of a mind fit the data equally, MOR (Def 4.6) selects the smallest description length — measured especially at the metacognitive level (how you model peers’ epistemics, Remark 4.7) — operationalized via the Real Log Canonical Threshold (RLCT) λ of Singular Learning Theory.
- The geometry. A broad ridge (low λ) carries more posterior mass than a tall narrow peak — prefer it.
- The mechanism. In every example (echo chamber 4.8, cult 4.9, pharma 4.10), the flattering hypothesis needs a fresh free parameter per disconfirming observation → description length grows without bound, while the simpler hypothesis is fixed-cost → MOR prefers it with growing confidence. Conjecture 4.11: pre-registration (ex-ante) closes the post-hoc patch channel.
Proven vs. open. Theorem (Watanabe 2009): the Bayesian free energy is asymptotically n·Lₙ + λ·log n — λ replaces BIC’s (d/2)log n, so Bayesian selection provably prefers low-λ solutions. Open (Tianyi Open Question 4.14): whether neural-net training realizes enough MOR via RLCT-induced inductive bias “for free,” or an explicit regularizer is required (Rec 5.3 lists candidates: weight-norm, dropout, tempering, SGLD, metacognitive-marginal penalties). Conjectural (Conjecture 4.11): that the true explanation sits on the low-λ subspace.
E. Active identification vs. MOR — complementary, not competing
The reward is underdetermined by passive behavior (Ng & Russell 2000; Armstrong & Mindermann 2018). Two distinct remedies:
- Active identification gathers data to resolve ambiguity that data can resolve. Guarantee — Zhonghao’s Identifiability Proposition: under (I1) sufficient belief correction, (I2) oracle-pass, (I3) comprehensive / Fisher-information-positive-definite active queries, the posterior converges in TV to a Dirac at
s*pref. Engine: active learning / optimal experimental design (Sadigh, Dragan, Sastry & Seshia 2017; Bıyık & Sadigh). Boundary — Zhonghao’s Impossibility-under-Akrasia: when behavior reliably mistracks the endorsed preference, no query separates “wants X, akratically does Y” from “wants Y.” So it identifies the acted-upon preference, conditional on behavior tracking it. - MOR is the tie-break where data cannot resolve the ambiguity — observationally singular states (Def 4.1: no observation distinguishes them) — so you fall back on the low-λ inductive bias.
They compose: active-ID up to the identifiability frontier, MOR beyond it. Neither is a blanket guarantee.
References
- R. Jacq, M. Geist, A. Paiva, O. Pietquin. Learning from a Learner. ICML 2019.
- M. Carroll, D. Hadfield-Menell, S. Russell, A. Dragan. Estimating and Penalizing Induced Preference Shifts in Recommender Systems. ICML 2022.
- N. Rabinowitz et al. Machine Theory of Mind. ICML 2018.
- D. Hadfield-Menell, S. Russell, P. Abbeel, A. Dragan. Cooperative Inverse Reinforcement Learning. NeurIPS 2016.
- A. Ng, S. Russell. Algorithms for Inverse Reinforcement Learning. ICML 2000.
- S. Armstrong, S. Mindermann. Occam’s razor is insufficient to infer the preferences of irrational agents. NeurIPS 2018.
- D. Sadigh, A. Dragan, S. Sastry, S. Seshia. Active Preference-Based Learning of Reward Functions. RSS 2017.
- P. Christiano et al. Deep Reinforcement Learning from Human Preferences. NeurIPS 2017.
- S. Watanabe. Algebraic Geometry and Statistical Learning Theory. Cambridge Univ. Press, 2009.
- R. Rao, D. Ballard. Predictive coding in the visual cortex. Nature Neuroscience, 1999.
- D. Ha, J. Schmidhuber. World Models. NeurIPS 2018.
- T. A. Qiu. Learning From Learners via Temporal Autoregression and the Metacognitive Occam’s Razor (
tianyi-may-lfl.md, working draft, 2026). - Z. He et al. Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning (2026).
F. Parked open problems (kept off the poster for now — 2026-06-06)
Removed from the poster’s §4 to keep it focused, recorded here:
- Akrasia / behavior-bypassing. Active identification recovers the acted-upon preference; under persistent
θ̂ ≠ θ*(reliable weakness of will) no query separates “wants X, akratically does Y” from “wants Y.” Needs a reflective-endorsement channel that doesn’t route through the akratic action. (Zhonghao’s Impossibility-under-Akrasia Proposition.) - Self-reference of the predictive system 𝒫. 𝒫‘s forecast of future preferences depends on the human’s future preferences, which depend on responses to 𝒫‘s forecasts — a fixed point to pin down.
Active-identification framing note: the poster now states active identification identifies the true latent belief (via active probing — present evidence, observe the update). This is distinct from the formal Identifiability Proposition, which is about the ideal preference identified via pair-queries under corrected beliefs. Clean reading: active probing → belief + update-process; elicitation under corrected beliefs → preference. (To reconcile §E above with the poster, treat them as two stages.)
Diagram gallery
All candidate poster illustrations in one place (3 also appear inline in Part 3 above, where they illustrate the text). Each notes its proposed poster section.
Confounded behavior · §1
Two latent causes → one behavior; watching cannot separate them.
State factorization · §2
S = S_is × S_belief × S_pref, with the direction-of-fit asymmetry.
Active identification → true latent belief · §3
The 2×2 metacognitive space · §3
Metacognitive Occam’s razor — ridge vs peak (SLT geometry) · §3
Metacognitive Occam’s razor — description length (mechanism) · §3
Path dependence · §4
