Overview

This directory is the formal write-up of the conjectured LFL Theorem, the central theoretical goal of the Learning From a Human Learner (LFL) agenda. It contains the model, the four pillar conditions, the theorem statement, and the proof sketch, split into pages that are meant to be read in order.

The problem

An AI assistant interacts with a human. The human is a learner. It holds partial information about the world and revises its beliefs as evidence arrives. We want to choose, and eventually train for, the assistant’s behavior, so that the combined human-assistant system serves the human as well as an ideal rational agent standing in the human’s place would serve itself.

The central difficulty is that the two parties are different agents. What counts as rational for the human depends on the environment, the assistant is part of that environment, so changing the assistant changes what counts as rational for the human.

The strategy

Full rationality is a package of distinct abilities. A rational agent must not hold and act on beliefs that are implausible relative to its own other beliefs and observations, it must exclude beliefs that fit the evidence only on a negligibly small set of configurations, and it must gather the further evidence that matters. The plan is to factor this package into four pillars. Each pillar names an objective that its own optimization process can target, together with the condition that holds when the optimization succeeds, and the theorem states that the four conditions jointly imply near-optimality of the combined system.

Two modeling devices connect the pillars. The human is modeled as a Gibbs planner. It computes the value of each action under some belief and chooses each action with probability exponentially increasing in that value, sharper or noisier according to a fixed rationality parameter, so behavior is stochastic but belief-driven. And each pillar’s condition is a budget, an upper bound on a divergence between the belief an agent acts on and the Bayesian posterior computed from everything that agent has seen. The divergence punishes implausibility only, never lack of coverage. Committing to one of the many views the evidence leaves open costs nothing, while mass placed on what the evidence weighs against grows expensive as the evidence accumulates.

1. Bounded Incoherence. The human never holds and acts on a belief that is implausible relative to its own other beliefs and observations. Pillar 1 bounds this internal incoherence by minimizing the divergence between the belief the human acts on and the human’s own Bayesian posterior, measured from the assistant’s vantage point; the divergence punishes implausibility only, committing to one of the many views the evidence leaves open costs nothing. (Provisional algorithm. Gibbs sampling?)
2. Controllability. Assuming Pillar 1, the human can control the assistant well enough that the assistant, too, behaves as an evidence-bounded planner, planning from its own posterior in the service of the human’s reward. We call the human and assistant together the joint agent. (Provisional algorithm. RLHF?)
3. Occam’s Razor. Assuming Pillars 1 and 2, the joint agent’s operative beliefs survive perturbation. Among beliefs that fit the evidence equally well, those whose fit is destroyed by a small perturbation of the belief are excluded. This closes the one permission the first two pillars leave open, they price implausibility by the evidence’s local density, and the razor makes the belief answer to the evidence’s mass. The razor’s essential domain is the metacognitive part of the state, the coordinates recording how each agent’s observations relate to the world, where evidence alone can never settle a disagreement. (Provisional algorithm. LLC-style perturbation sampling?)
4. Action Uncertainty Resolution. Assuming Pillars 1 through 3, the pair selects actions whose observations discriminate between the hypotheses that affect its behavior, so that no achievable resolution of its own future behavior is left unrealized. The first three pillars govern how the pair’s belief answers to its evidence; this one makes the evidence itself decisive where decisions differ. (Provisional algorithm. Collaborative forecasting with log scoring?)

The four gear together in one chain. Pillar 4 makes the pooled posterior concentrate wherever the pair’s decision depends on the answer. Pillars 1 through 3 force the belief the pair acts on to sit where that posterior puts its mass. And a Gibbs planner holding a belief that sits there takes the action that is best there, which is what near-optimality means.

Remove any link and the chain breaks. Without Pillar 4 the posterior stays diffuse and there is nothing to force the belief onto. Without Pillars 1 and 2 the pair may act on a belief its own evidence contradicts. Without Pillar 3 the belief may sit on a set of high density and negligible mass, cheap by the earlier budgets’ accounting and wrong about where the evidence points.

The four conditions in sequence. Each pillar strengthens the guarantee established by the ones before it, and the final step is the composition laid out on the proof-sketch page.

---
config:
  markdownAutoWrap: false
  flowchart:
    curve: linear
    wrappingWidth: 450
  themeVariables:
    fontSize: 14px
---
flowchart TD
  L0["Two separate agents, no guarantee on the outcome"]

  subgraph SG1[" "]
    direction LR
    P1["<b>Pillar 1 (via Gibbs sampling?):</b><br/>Bounded Incoherence"] -.-> P1X["<b>Pillar 1 excluded:</b> self-contradictory<br/>or observation-contradictory beliefs"]
  end
  L0 --> SG1
  SG1 --> L1["<b>Learner divergence-bounded:</b><br/>acts upon a belief that's mostly plausible but possibly assumptive"]

  subgraph SG2[" "]
    direction LR
    P2["<b>Pillar 2 (via RLHF?):</b><br/>Controllability"] -.-> P2X["<b>Pillar 2 excluded:</b> assistant manufacturing<br/>needless uncertainty against learner"]
  end
  L1 --> SG2
  SG2 --> L2["<b>Assistant divergence-bounded for the learner's reward:</b><br/>the pair acts as two evidence-bounded planners serving the learner"]

  subgraph SG3[" "]
    direction LR
    P3["<b>Pillar 3 (via LLC?):</b><br/>Occam's Razor"] -.-> P3X["<b>Pillar 3 excluded:</b> inventing convoluted<br/>justifications for false beliefs"]
  end
  L2 --> SG3
  SG3 --> L3["<b>Joint agent's beliefs survive perturbation:</b><br/>belief must sit where the evidence leaves mass, not merely density"]

  subgraph SG4[" "]
    direction LR
    P4["<b>Pillar 4 (via collaborative forecasting?):</b><br/>Action Uncertainty Resolution"] -.-> P4X["<b>Pillar 4 excluded:</b> assistant allowing<br/>learner to believe what they want,<br/>not seeking counterevidence"]
  end
  L3 --> SG4
  SG4 --> L4["<b>Posteriors concentrated wherever it matters:</b><br/>the evidence now decides where the belief is forced to sit"]

  L4 --> GOAL["<b>Conjectured LFL Theorem:</b> joint agent matches the idealized<br/>pooled-information team up to ε in average reward"]

  classDef pillar fill:#d98b78,stroke:#a35f4e;
  classDef excl stroke-dasharray: 5 5;
  classDef goal fill:#2b6cb055,stroke:#2b6cb0;
  class P1,P2,P3,P4 pillar;
  class P1X,P2X,P3X,P4X excl;
  class GOAL goal;
  style SG1 fill:transparent,stroke:none
  style SG2 fill:transparent,stroke:none
  style SG3 fill:transparent,stroke:none
  style SG4 fill:transparent,stroke:none

Reading the diagram. Each red box names a pillar together with the provisional algorithm expected to implement it, if known. The dashed arrow out of each red box points to the characteristic failure that pillar newly excludes, and the white box below states the property that holds once the pillar is in place on top of the previous ones.

The theorem, informally

Imagine an idealized version of the team, the same human and the same assistant, except that each member sees everything the two have jointly seen, and each weighs that shared evidence exactly correctly. The idealized team receives no outside knowledge, it only makes perfect use of the evidence the actual team already has.

The conjectured theorem says that a team satisfying the four conditions serves the human nearly as well as this idealized team would, and that “nearly” tightens toward “exactly” as the four conditions are met more and more precisely. Once nothing is lost to incoherence, to the assistant working against the human, to beliefs that fit only by accident, and to evidence never gathered, there is nothing left to lose. The exact construction of the idealized team, and the exact sense of “nearly as well”, are in the theorem statement.

What is modeled and what is not

We do not model the process of policy optimization explicitly. These pages study the end states such a process can reach. When certain self-supervised losses are decreased below a threshold, one per pillar, the pair’s shortfall against the idealized team admits a vanishingly tight bound. Whether training reaches those end states, and how the conditions are estimated in practice, is recorded in the training and gap sections at the end of the pillar pages and the theorem statement.

Reading guide

• Basic Formalism. The decision-process model, the single-agent machinery, and the regularity conditions everything else assumes.
• Pillar 1 Definitions. Gibbs rationalization, the belief divergence, and Condition 1.
• Pillar 2 Definitions. The pooled posterior, the joint belief divergence, Condition 2, and this pillar’s theoretical and empirical gaps.
• Pillar 3 Definitions. The perturbed divergence, Condition 3, and this pillar’s theoretical and empirical gaps.
• Pillar 4 Definitions. Continuations, per-target resolution, record-carried information, and Condition 4.
• LFL Theorem Statement. The reference process, the gap, the regular instance, and the theorem.
• LFL Theorem Proof Sketch. Six lemmas and two propositions, with the remaining gaps listed at the end.
• Glossary of Notation. Every symbol, with the page and block where it is defined.
• What’s Next?. The map of the open gaps, and the questions that extend the framework.