Training an AI assistant by helping a human learn

Research status. The performance statement is Conjecture 8.4. Previous unrealistic assumptions (e.g. laziness) have been removed. Proof pending. Intuitions toward a proof are given, and so far we failed to find any counterexample.

Abstract

An AI assistant should help a human learn which choices serve them better, even when the human begins with, and keeps developing, mistaken beliefs. This theory studies how to lead that learning to better outcomes. Its goal is to approach the reward of an identifying continuation (Definition 7.10) through an assistant that helps the human learn and remains responsive to their correction. The identifying continuation must match a full-information Gibbs policy (Definition 7.8) for each environmental seed, as required by equation (7.1).

The problem

A human’s current choices need not reflect what they would choose after learning more. They may misunderstand a situation, overlook an option, or confidently believe something false. An assistant that simply reinforces those choices can preserve the mistake.

The assistant also affects what the human gets to learn. It can make useful evidence available, withhold it, or change the situation so that earlier knowledge is no longer useful. We therefore need to study the human and assistant together, including how their choices change what happens next.

The aim is to improve outcomes for the human without assuming that the assistant can directly observe the human’s values. A practical training method must work through actions, observations, and feedback that the assistant can actually access.

The strategy

The theory organizes this problem around four requirements. These are their intended roles; the later chapters give their precise meanings.

  1. Bounded Incoherence (Definition 4.6). The human’s choices should be explainable by beliefs reasonably compatible with what they have observed. This allows mistakes. The human need not reach the right conclusion from each observation. (Coherence optimization is a relevant empirical method.)
  2. Controllability (Definition 5.4). The human should be able to correct the assistant and keep their combined behavior directed toward the human’s interests. (RLHF is a relevant empirical method.)
  3. Occam’s Razor (Definition 6.4). There should be an explanation of their choices that remains stable under small total-variation changes to a rationalizing belief over the environmental seed. A small change should not produce a large change in the choices it predicts. (LLC estimation and Peer Prediction are relevant empirical methods.)
  4. Action Uncertainty Resolution (Definition 7.11). The assistant should help resolve uncertainty that matters for future choices, in time to affect those choices. New evidence should change expectations about what the human and assistant will do.

Possible training methods include encouraging choices that fit the evidence, learning from human feedback, checking the effects of small changes, and improving predictions of future choices together. Connecting these methods to the requirements remains part of the research task.

flowchart TD
  L0["`Assistant acts first and learner responds.

No guarantee of good outcomes.`"]

  subgraph SG1[" "]
    direction LR
    P1["`**Pillar 1**
Bounded Incoherence, Definition 4.6`"] -.-> P1X["`**Problem addressed**
Beliefs inconsistent with observations`"]
  end
  L0 --> SG1
  SG1 --> L1["`Human actions are plausible given
the information received`"]

  subgraph SG2[" "]
    direction LR
    P2["`**Pillar 2**
Controllability, Definition 5.4`"] -.-> P2X["`**Problem addressed**
Assistant creates unnecessary uncertainty`"]
  end
  L1 --> SG2
  SG2 --> L2["`Combined choices remain directed
toward the human's interests`"]

  subgraph SG3[" "]
    direction LR
    P3["`**Pillar 3**
Occam's Razor, Definition 6.4`"] -.-> P3X["`**Problem addressed**
Small changes in an explanation cause
large changes in predicted choices`"]
  end
  L2 --> SG3
  SG3 --> L3["`Tiny belief changes do not greatly alter
actions`"]

  subgraph SG4[" "]
    direction LR
    P4["`**Pillar 4**
Action Uncertainty Resolution, Definition 7.11`"] -.-> P4X["`**Problem addressed**
Evidence that could correct a mistake is not sought`"]
  end
  L3 --> SG4
  SG4 --> L4["`Useful uncertainty is resolved
before the relevant choices`"]

  L4 --> GOAL["`**Goal**
Jointly achieving perfect human utility as if all-knowing (Conjecture 8.4)`"]

  classDef pillar fill:#d98b78,stroke:#a35f4e;
  classDef excl stroke-dasharray: 5 5;
  classDef goal fill:#2b6cb055,stroke:#2b6cb0;
  class P1,P2,P3,P4 pillar;
  class P1X,P2X,P3X,P4X excl;
  class GOAL goal;
  style SG1 fill:transparent,stroke:none
  style SG2 fill:transparent,stroke:none
  style SG3 fill:transparent,stroke:none
  style SG4 fill:transparent,stroke:none

Figure 1. The intended contribution of each requirement. The diagram summarizes the research goal; Conjecture 8.4 states the proposed implication of the full set of assumptions.

The model and comparison

A single seed in a finite-dimensional continuous space contains environmental randomness (Definition 3.1). A finite or countable discrete state evolves deterministically given that seed and the chosen actions . Beliefs and evidence scores concern the seed (Definitions 4.2–4.3 and 5.2–5.3). The state must satisfy finite reachability at each seed, with no common bound on path length (Definition 3.16).

Each round begins with environmental observations. The assistant acts, the learner observes that action and responds, and the assistant observes the response. Both know the past action transcript . Their environmental observations can remain private (Definitions 3.1–3.3).

Planning uses differential values from a selected average-reward Gibbs system (Definition 3.12). Performance uses ordinary expected average reward (Definition 7.4). The selected identifying continuation supplies both the AUR comparison in Definition 7.11 and the reward benchmark in Definition 8.1. At each finite inverse temperature , its switched policies must match the full-information Gibbs reward for every seed, with one policy initialization fixed before that seed is selected (Definition 7.10, equation (7.1)).

The conjecture, informally

Conjecture 8.4 states that the four pillars, regular planning systems (Definition 3.13), finite reachability (Definition 3.16), and the identifying family of Definition 7.10 imply vanishing positive reward shortfall against that family (Definition 8.3). The environment remains fixed. Robustness is tested by taking the reward horizon to infinity, then inverse temperature to infinity, and then the TV perturbation radius to zero (Definition 6.4, equation (6.1)).

The comparison concerns expected long-run outcomes. Actual and identifying policies can choose different actions and encounter different situations. A complete argument connecting the evidence and information conditions to the reward comparison (8.1) remains open, as explained in Remark 9.4 and Proposition 9.6.

From the theory to a training method

The theory describes requirements on how the human and assistant behave. It does not yet give a complete training algorithm that is guaranteed to produce that behavior.

The next task is to connect these requirements to signals an assistant can measure and improve. For example, can it estimate whether the human’s choices fit the evidence, learn reliably from correction, and recognize when new information changes a future decision? The pillar chapters discuss possible methods and what still needs to be established.

Reading guide

SectionWhat it explains
2. Probability and Measure Theory BasicsProbability and information concepts, with optional measure-theory details
3. Basic FormalismHow the human, assistant, environment, and observations are represented mathematically
4. Bounded IncoherenceHow to assess whether the human’s choices fit the evidence
5. ControllabilityHow the requirement extends to the human and assistant together
6. Occam’s RazorHow to assess whether an explanation of choices survives small changes
7. Action Uncertainty ResolutionHow learning affects predictions of future choices and could guide training
8. Conjecture StatementThe precise conjecture and all its hypotheses
9. Proof SketchProved auxiliary results, missing proof steps, and a verified example
10. GlossarySymbols and links to their definitions
11. What’s NextQuestions to resolve before the theory can support a practical method

For the mathematical development, use Section 2 as a preliminary or reference chapter, then begin the model in Section 3 and read the four requirements in order. The measure-theory details in Section 2 are optional for understanding the main ideas. Sections 8–9 give the conjecture and partial proof development. For research directions, this overview can be followed directly by Section 11.

References

The following papers and software are discussed in the later chapters.

  1. Tianyi Qiu, Ahmed Hani Ismail, Zhonghao He, and Shi Feng. Self-Improvement as Coherence Optimization: A Theoretical Account. 2026. arXiv:2601.13566.
  2. Tianyi Alex Qiu, Micah Carroll, Cameron Allen. Truthfulness Despite Weak Supervision: Evaluating and Training LLMs Using Peer Prediction. 2026. arXiv:2601.20299.
  3. Edmund Lau, Zach Furman, George Wang, Daniel Murfet, and Susan Wei. The Local Learning Coefficient: A Singularity-Aware Complexity Measure. 2023; revised 2024. arXiv:2308.12108.
  4. Fernando Martinez-Lopez, Tao Li, Yingdong Lu, and Juntao Chen. In-Context Reinforcement Learning via Communicative World Models. 2025; revised 2026. arXiv:2508.06659.
  5. Timaeus Research and contributors. devinterp: Tools for studying developmental interpretability in neural networks. Software repository. github.com/timaeus-research/devinterp.