LFBL / MAB — Meeting Notes (2026-07-18)

Meeting on the multi-armed-bandit (MAB) “learning from a biased demonstrator” framework (part of Learning-from-Learner Theory). Companion to the working note lfbl-mba-in-progress. Lucas was out for the July 19 session — async follow-up via Slack + Hub.

At a glance

flowchart TD
  A["Biased demonstrator (B-UCB): prior anchor ν, intensity λ"]
  A --> P["Passive: observe action history H_T only"]
  A --> I["Active: intervene on the environment"]
  P --> PE["Observationally equivalent to a rational agent with reward μ+b"]
  PE --> PF["Le Cam two-point: constant error floor, no improvement with T"]
  I --> S["Stakes manipulation γ: estimate ξ1=μ+b and ξ2=γμ+b"]
  S --> SR["Isolate μ = (ξ2 − ξ1)/(γ−1) → O(1/√T)"]
  I --> H["Horizon truncation: collapse exploration bonus → pure exploitation"]
  SR --> OPEN["Open: uniform-coverage assumption fails when λ ≫ T"]
  A --> T["Tianyi: constant λ washes out as pulls grow → only transient bias captured"]
  T --> B["Proposed: broader parameterized policy space (belief → action map)"]

Action items

• Lucas Biechy — Update the Hub with revised theory + formalism: incorporate meeting feedback — improve the upper-bound proofs, consider Tianyi’s broader parameterized policy space, and note the open uniform-coverage assumption in the strong-bias regime.
• Tianyi Alex Qiu — Send Lucas a calendar invite for a follow-up session (Lucas is out tomorrow).
• Yuxiao Li — Review Lucas’s write-up more thoroughly and share thoughts on the upper bound — consider self-normalized concentration inequalities and adaptive-data bounds as alternatives to the current derivation.

Overview

• Lucas built out the MAB framework for learning from a biased demonstrator on the Hub, extending the inverse-bandit paper Yuxiao shared.
• The passive lower bound (Le Cam two-point) is considered sound; the interventional upper bound needs more work — Yuxiao thinks a self-normalized concentration approach may be the right path.
• Tianyi’s main concern: the B-UCB bias washes out over time, so the model captures only biases affecting intermediate behavior, not long-run convergence — he suggested a broader parameterized policy space.
• Lucas unavailable for the July 19 meeting; async updates via Slack + Hub.

Biased-demonstrator model (B-UCB)

Bias is modeled by adding a prior anchor vector $\nu$ and pseudo-count intensity $\lambda$ to the reward estimate, giving a regularized empirical mean that blends observed rewards with the prior:

$${\tilde{\mu }}_i(t)=\frac{{\sum }_{\tau =1}^t\mathbb{I}(A_{\tau }=i)Y_{\tau }+{\lambda }_i{\nu }_i}{N_i(t)+{\lambda }_i}$$

The resulting B-UCB policy picks the arm maximizing the biased index, where the exploration bonus decays at $1/\sqrt{N_i(t)+{\lambda }_i}$ rather than the standard $1/\sqrt{N_i(t)}$.

Tianyi's concern (transient vs. permanent bias)

Because $\lambda$ is constant while pull counts grow roughly linearly, the bias terms are diluted over time — the biased and rational policies converge to the same arm distribution given enough data. So the current model captures transient, not permanent, bias.

Counterexamples — bias vs. rationality equivalence

Two cases where a biased and an unbiased agent are observationally equivalent:

1. Additive perception bias $b_i$ (exact). With symmetric noise, a biased agent perceiving ${\mu }_i+b_i$ is indistinguishable from a rational agent whose true reward is ${\mu }_i+b_i$.
2. Bayesian prior bias $(\nu ,\lambda )$ (asymptotic only). At finite $t$ the exploration bonuses differ ($\sqrt{N_i(t)+{\lambda }_i}$ vs $\sqrt{N_i(t)}$), so the agents are technically distinguishable from their exploration patterns. Lucas flagged he’s not fully confident in this one.

Interventions to break observational equivalence

• Stakes manipulation (double rewards for a phase): a fixed internal bias $b_i$ doesn’t scale with the payout multiplier, but a true reward preference does — comparing how arm-choice probability shifts isolates the linear weight of $b_i$.
• Horizon truncation (announce termination at $T=t+1$): collapsing the exploration bonus to zero forces pure exploitation — a prior-belief-biased agent drops a bad arm once its empirical mean has been dragged down, while an intrinsic-utility-biased agent keeps pulling it regardless.

Passive lower bound & interventional upper bound

Passive minimax lower bound (Le Cam two-point): two instances with identical effective means $\mu +b$ but different true $\mu$ produce indistinguishable history distributions, so no estimator beats a constant error floor regardless of $T$. Yuxiao considers this sound.

Interventional upper bound: partition the horizon into two phases ($\gamma =1$ and $\gamma >1$), estimate ${\xi }^{(1)}=\mu +b$ and ${\xi }^{(2)}=\gamma \mu +b$ separately, then isolate

$$\hat{\mu }=\frac{{\xi }^{(2)}-{\xi }^{(1)}}{\gamma -1},\text{error }\mathcal{O}(1/\sqrt{T}).$$

Open assumption (flagged by Lucas)

The upper bound relies on uniform coverage ($\mathbb{E}[N_i(T/2)]\ge cT$), which conflicts with the strong-bias regime $\lambda \gg T$ where biased arms are chronically under-pulled. Yuxiao thinks it can be salvaged but likely needs self-normalized concentration inequalities rather than standard Hoeffding, given the adaptively collected data.

Broader parameterized policy space (Tianyi)

Move beyond UCB-specific bias to a larger policy class: the agent maintains a Gaussian belief over each arm’s reward, and the policy is a parameterized map from that belief vector to an action distribution.

• Motivation: real-world biases often work by causing agents to act on misinformed beliefs and never collect the data that would correct them — an action-dependent observation signal the current formalism doesn’t capture.
• Tianyi wants to redo the passive + interventional bounds over this broader class, unifying Lucas’s current formalism with more general action-dependent bias.

Connection to Julius’s work & LLMs

• Lucas’s concern: the MAB framing may be too disconnected from Julius’s work to collaborate, unlike the MDP framing which maps more naturally to how RL works in LLMs.
• Tianyi: the core insight — passive observation is fundamentally worse than intervention for inverse inference — carries over to the LLM case, and the interventional approach can likely be mapped to a language-model training objective.
• Yuxiao: Julius is benchmarking RLHF with human feedback (similar bias questions); suggested writing up shared assumptions/findings to integrate the two threads.

Related

lfbl-mba-in-progress · lfbl-mdp-in-progress · Learning-from-Learner Theory