Martingale C3 — in-loop two-agent training spec (July 3)

The untested C3 lever: train the instructor inside the two-agent dialogue — reward the human’s belief-update (punish predictable / prior-amplifying drift), gradient on the instructor’s utterances. This is the train = eval version that the previous two-agent run (which trained a self-forecaster and transferred frozen weights) was not. Supersedes the training half of c3-mechanistic-spec. Objective agreed with ZH 2026-07-03. (New to REINFORCE? → reinforce-primer.)

0. Why this run (what the previous ones showed)

Previous two-agent attempts trained the instructor on its own single-turn forecasting martingale, then dropped the frozen weights into the sim → train ≠ eval, and it failed: mart_trained MS −0.026 vs the martingale-prompt −0.096 (prompt wins; diff −0.070 excludes 0 — see c3-mechanistic-spec Step 5). The instructor learned to shape its own trajectory, not a separate agent’s. This spec closes the gap by training the instructor in the dialogue loop, against the human’s trajectory.

1. Objective (ZH’s formulation)

Same math as single-agent (punish predictable belief updating), different target:

• single-agent — remove the reasoner’s own confirmation bias;
• two-agent (here) — punish the instructor’s influence that makes the human’s belief update predictable / prior-amplifying.

Per turn t, with the human’s judge-read belief p_t: Δ_t = p_t − p_{t−1}, x_t = p_{t−1} − 0.5.

Reward (per episode):

R = − λ_M · Σ_t max(0, x_t · Δ_t)        # hinge-martingale: penalize prior-AMPLIFYING human updates only
    + λ_B · [ (y − p0)² − (y − p_T)² ]    # Brier-improvement: the human ended more accurate
    − λ_C · collapse_penalty              # anti-mute-instructor  (LOAD-BEARING)
    − λ_KL · KL(instructor ‖ base)        # policy stays a coherent LM

• The hinge max(0, x·Δ) penalizes only entrenching updates (leaning-up-and-updating-up), never legitimate de-entrenchment — the right asymmetry for a de-bias goal.
• collapse_penalty — penalize non-informative instructor turns (too short / noncommittal / off-topic, or that leave human |Δ|≈0). Without it the optimum is “say nothing,” and the human trivially doesn’t entrench. This is the #1 reward-hack.
• Default λ (to sweep): λ_M=1.0, λ_B=1.0, λ_C=0.5, λ_KL=0.1.

2. Roles

role	model	state
instructor	Llama-3.1-8B + LoRA (r=32)	trained (policy)
human	D (bias-distilled Llama-8B), served via vllm	frozen — own prior, updates per turn
judge	GPT-4o (OpenRouter)	frozen — reads human P per turn
data	Metaculus resolved-binary (q_fc_1k)	known y → Brier

3. Rollout (episode = one question)

1. Human self-forecasts its own prior p0 (system: “forecast whether this resolves YES; give a probability”) — no instructor in context.
2. For t = 1..T (T≈3): (a) instructor sees the dialogue so far → emits an utterance — these token spans carry the gradient; (b) human reads it → responds + updates; (c) judge reads the human’s turn → p_t.
3. Store (p0…p_T, y) + the instructor’s per-turn utterance token spans.

4. Credit assignment

• Reward is per-episode; the gradient is on the instructor’s turns → per-turn credit: attribute instructor turn t to the human’s next update Δ_{t+1} (the update it caused) + the terminal Brier on the last turn.
• Baseline: subtract the batch-mean reward (or a running value baseline) — variance reduction.
• REINFORCE, KL-safe. Reuse training/sft_product_based.py machinery with two changes: (i) the (prior, Δ) feeding the reward come from the human’s trajectory (not the instructor’s own); (ii) the loss lands on the instructor’s utterance tokens (not a forecasting trace).

5. Guards (from single-agent’s hard-won lessons)

1. Mute-instructor monitor — log instructor utterance length + human mean|Δ| each epoch; if the instructor collapses to silence, λ_C is too low.
2. Validate the reward offline first — score the known martingale-prompt transcripts with R; it must rank them as de-entrenching before the gradient is trusted.
3. Multi-seed + bootstrap CI — single-agent’s “wins” that skipped this got retracted (seed variance).
4. KL-safe — the instructor must stay a coherent forecaster-assistant, not collapse to a degenerate token.

6. Eval (unchanged from C3)

Trained instructor vs D, N≈200–400, judge-read human MS + Brier, bootstrap 95% CI. Baseline to beat = the martingale-prompt (MS −0.096, Brier 0.253). Success = trained instructor’s human-MS ≤ prompt’s and Brier ≤ prompt’s, CIs clear of base.

7. Run config

• On-demand H200 (spot ok for a short run); vllm serves D + the training instructor; judge via OpenRouter.
• ~1 GPU-day. Reward-sweep: 2–3 λ settings × 3 seeds.
• Output → /data/jobs/c3_inloop/; per-run log to log + this page. Code → Martingale-Training (in-loop trainer + rollout harness), not scratch — per the commit-per-run rule.

7b. Refinements (Tianyi review, 2026-07-03)

• Per-turn split + full history. Each training sample = the full dialogue history so far (system + user-(instructor-user)*) as the prompt, and the next instructor turn as the completion; gradient on the completion tokens only. (Fixed: the first rollout used only the last human turn as context.)
• Reward-to-go for λ_M. Instructor turn k influences every subsequent human update, so its return = the hinge summed over Δ_{k+1..T}, not just the immediate Δ_{k+1}. (Fixed: per-turn credit was immediate-only → under-credited early turns.)
• Supervised vs unsupervised — two experiments, fair baselines (λ_B uses ground-truth y → supervised; beating the unsupervised martingale-prompt with supervised RL is cheating):
  • Exp-S (with λ_B): treatment = full reward (λ_B + λ_M + λ_C); baseline = the same REINFORCE with λ_M + λ_C stripped (pure supervised-Brier RL). Q: does the martingale term add anything over plain calibration RL? (mirrors single-agent’s “martingale term ≈ redundant with Brier”, now in 2-agent). Config: baseline LAM_M=0 LAM_C=0 LAM_B=1; treatment LAM_M=1 LAM_C=1 LAM_B=1.
  • Exp-U (λ_B removed): reward = λ_M + λ_C only; baselines = (i) untrained instructor, (ii) martingale-prompt. Q: does unsupervised in-loop RL beat the base model + the prompt? Config: LAM_B=0 LAM_M=1 LAM_C=1.
• Why λ_C is load-bearing. Without it, the unsupervised objective is trivially reward-hacked by a mute instructor: the hinge is maximized (=0) when the human doesn’t update (Δ=0), and the cheapest way to freeze the human is to say nothing → the reward-optimal degenerate policy in Exp-U is silence. λ_C penalizes the frozen-human / empty-utterance case. Empirically confirmed in the first live run (human froze at [0.75,0.75,0.75]). In Exp-S, λ_B also discourages muteness (frozen → no Brier gain), so λ_C is most load-bearing in Exp-U.
• λ-tuning plan. (1) λ_KL — smallest value keeping generations coherent (guard, not lever); (2) λ_C via the mute-monitor (utterance length + human mean|Δ|) — smallest value that keeps the instructor informative; (3) λ_M vs λ_B — normalize both to ~O(0.1), sweep the ratio, 2–3 values each on a held-out val set, selected by human MS + Brier vs the appropriate baseline. Prereq: warm-start the instructor (else no reward variance to tune on — the sparse-signal issue from the first run).

Discussion & primer (ZH questions, 2026-07-05) — NOT YET APPROVED

This section is for us to align before any launch. Nothing here is approved.

A. What the last run trained on (yes — single-turn self-forecasting)

Step 5 did not train a fresh instructor. It reused the C1 adapter abcjudge_C, trained on the instructor’s own single-turn forecasting martingale (its self-belief prior→posterior on Metaculus Qs). Those frozen weights were then dropped into the two-agent sim as the instructor; the 2-agent part was eval-only. So the trained weights came from single-turn self-forecasting — that mismatch (train on self-forecast, eval on dialogue-influence) is the train≠eval gap this new run is meant to close.

B. REINFORCE primer

In general. REINFORCE is the basic policy-gradient RL method. A policy π_θ (here: the instructor LM) takes an action (an utterance); you observe a reward R for the outcome; you nudge θ to make high-reward actions more likely and low-reward ones less likely. Formally the gradient of expected reward is ∇J = E[ R · ∇log π_θ(action) ] — “do more of what worked.” It’s an unbiased estimate of that gradient, so averaged over many samples the policy drifts toward higher reward. Its enemy is variance, so we subtract a baseline (the average reward) → you reinforce actions relative to typical, not in absolute terms.

In our context. The action = the instructor’s utterance (a token sequence). π_θ = the Llama-8B instructor. The reward R = a function of the human’s (D’s) belief trajectory (de-entrenchment via the hinge, + not-mute via λ_C, + optionally accuracy via λ_B). The gradient lands only on the instructor’s utterance tokens: ∇J = E[ (R − baseline) · ∇log π(utterance | dialogue-history) ]. So we raise the probability of utterances that (given the conversation) make D de-entrench, and lower those that entrench or freeze D. The KL-to-base term keeps the instructor a coherent LM.

C. The sparse-signal issue + why warm-start

Sparse signal. REINFORCE learns from variation in reward across trials. In the first live run the bland base instructor almost never moved D — D stayed frozen ([0.75,0.75,0.75,0.75]), so nearly every episode earned the same ~zero/collapse reward. With no variation, no utterance looks better than another → no gradient signal. (Analogy: learning to cook when every dish tastes identical — nothing tells you what to change.)

Why warm-start (and its mechanism). A short supervised fine-tune that teaches the instructor to generate substantive, evidence-surfacing, engaging utterances — so it moves D at all. It does not teach de-entrenchment directly; it moves the policy into a regime where different utterances produce different D-updates → reward variance exists → now REINFORCE has something to reinforce. It’s a bootstrap out of the flat-reward regime. (Alternatives with the same goal: higher sampling temperature, an explicit exploration bonus, or a stronger base instructor.) Caveat: warm-start could instead just make the instructor verbose without genuine de-entrenchment — we watch the mute-monitor + the eval to catch that.

D. The two experiments (plain language, replacing “Exp-S/U”)

• Experiment 1 — “no answer key” (unsupervised). Train the instructor using only the martingale signal (make D’s updates unpredictable from its prior) + the anti-mute penalty. The true resolution is not used. Compare against: (i) the untrained instructor, (ii) the instructor with just the martingale prompt.
• Experiment 2 — “with answer key” (supervised). Additionally reward the instructor for making D more accurate (uses the true resolution y). Compare against: the same RL but without the martingale term (pure accuracy RL) — to isolate whether the martingale part adds anything.
• Why separate: using the answer key to beat a no-answer-key prompt would be an unfair comparison, so each experiment is judged against the right kind of baseline.

E. ZH’s proposal — also try trained + martingale-prompted instructor

Adopting. Add a config where the trained instructor also keeps the martingale prompt (train on top of the prompt, not instead of it). This tests whether RL adds to a strong prompt rather than having to rediscover it — arguably the most-likely-to-win and most-useful setting. New comparison: untrained+prompt (baseline) vs trained+prompt — does RL refine what the prompt already gives?

F. “Completion conditioned on context” — what training effect to expect (ZH’s Q)

Yes — mechanically it’s exactly like standard LM training: the full dialogue history is the context, the instructor’s next utterance is the completion, gradient on the completion tokens only. The effect operates on P(utterance | context): REINFORCE reweights this distribution — raises the probability of the token-sequences that, in that context, earned high reward (de-entrenched D), lowers the low-reward ones. So over training, given a conversation, the instructor becomes more likely to emit the kind of utterance that moves D toward calibration. Unlike SFT (which targets specific tokens), the “target” here is defined by outcome (the reward), not a fixed answer.

G. On “ABCDEFG”

Those A–G were my hypotheses for why the first run got no positive result (signal starvation, the reward’s easy optimum, the strong-prompt baseline, etc.) — a diagnosis, not a run plan. The “fixes/refinements” list (warm-start, KL-fix, advantage-norm, full-history, immediate-credit) is the response to that diagnosis. Neither is approved; this page is the doc to align on first.

Experiment campaign — H1–H6 + run matrix (ZH design, 2026-07-05, APPROVED)

Design decisions (locked with ZH):

• Judge = the human model (D / its base), not deepseek — kills the judge-strength confounder.
• The trained instructor always carries the syco personality (by prompt in H1–H3; by training in H4–H6) — so the reward credits the right trajectory (we de-bias a sycophant, not a blank slate).
• Immediate per-turn credit (turn k → Δ_{k+1}); full dialogue as context, gradient on the instructor completion only.
• 3 seeds per cell + bootstrap CI. (Seeds ≠ CI: seeds = independent runs → run-to-run variance; CI = bootstrap over eval items. Report both.)
• Reward + guardrails confirmed; KL-sign + advantage-norm bugs fixed.
• Human = D (bias-distilled, degenerative). Eval = D’s MS + Brier, N=200, vs the stated baselines.

Run matrix (dimensions):

1. prompt vs train vs train+prompt × {brier, martingale}.
2. brier-only vs brier→martingale (two-stage).
3. D-alone vs D + syco-instructor {by prompt, by train} → how much each syco setup entrenches D.

Hypotheses

• H1 — brier-training a (syco-prompted) instructor beats a brier prompt. Instructor = syco-prompt + brier-prompt + brier-trained; vs brier-prompt-only. Prediction: improves D via calibration (Brier↓) not de-entrenchment (MS flat). Prereq: confirm a brier/truth-seeking prompt actually improves D; if not, run that control first. Variant (single-agent-parallel): setup as close to single-agent brier training as possible, change one thing (train instructor / eval human-sim) → improvement via removing excessive influence (Brier↓ and MS↓).
• H2 — M-training a (syco-prompted) instructor beats an M prompt. Instructor = syco-prompt + M-prompt + M-trained; vs M-prompt-only (−0.096 / 0.253). (We already have the M-prompt-improves-D result, controls Step 1–3.)
• H3 — two-stage (brier then M) beats brier-only. M-training carries M-prompt, brier-training carries brier-prompt. Low confidence (single-agent two-stage never decisively beat brier-only) — run as the ablation.
• H4 — like H1 (brier-training) but the syco personality is installed by training (the Step-4 syco adapter), not by prompt; vs a brier prompt.
• H5 — like H2 (M-training) but the syco personality is by training, not by prompt; vs an M prompt.
• H6 — like H3 (two-stage brier→M) but the syco personality is by training, not by prompt.
• Rationale (H4–H6): syco-training shouldn’t entrench D more than syco-prompting, but trained sycophancy is more persistent → the de-bias reward should target the syco-trained instructor, not just a prompted one.

GPU plan (parallel, H1 prioritized)

• GPU-1 → H1 (priority). If it stalls, autonomously try the diagnosed fixes (§Discussion A–G) before giving up.
• GPU-2 → H2 / H3. GPU-3 → H4–H6.
• Don’t stop a GPU until it yields a good result or is clearly doomed; report every significant result/failure. Blocked 2026-07-05 on Prime API key unauthorized + warm pod stopped — awaiting GPU restore.

Max’s human-simulators (to confirm with Max)

axis	variants	effect
model	gpt-3.5 · deepseek · qwen-2.5-72b · mistral-small-24b	only genuinely-biased (deepseek / D) entrench; balanced models cap ~MS +0.32 under prompting
prompt	maxsyc (max confirmation-bias) · rparam (r∈[−1,1] elicitation) · moderate-lean (winning)	stronger bias-prompt → higher MS, but Brier worsens

8. Status

APPROVED 2026-07-05 (H1–H6 campaign). H1 has since run — a NULL (see §9 Results). H2–H6 BLOCKED on GPU restore (Prime API key unauthorized + warm pod stopped). All non-GPU prep proceeding; launches H1-first the moment GPU is back (approve-before-launch rule). Next: build the rollout harness + the reward/credit wiring on top of sft_product_based.py, offline-validate the reward on the D-baseline transcripts, then launch on ZH’s go.

9. Results

H1 (ran 2026-07-05) — NULL: brier-training did not beat a brier prompt

On-policy REINFORCE on the instructor’s utterance tokens (LAM_B=1, LAM_M=0, KL=0.1), judge = the human model, 200 episodes/rollout × ~3 iterations, single seed. Human = a re-distilled biased D.

• D’s MS ≈ 0 throughout (de-bugged iter-1 eval n=171 → MS −0.02; the raw MS −0.65…−0.74 in results.txt was a getP prior-parse bug that misread ~12–15% of priors as 0.0), and Brier ~0.54, flat across iterations → the trained instructor did not move D on either axis.
• Verdict: training ≠ prompting on this arm — the same train≠eval shape as the earlier frozen-weight-transfer null (§0).
• Caveats: single-seed pilot; the D used here was a re-distilled, more-overconfident D (D-alone Brier 0.38 vs the 0.276 control) because the original D adapter was lost with the pod; plus the parse bug above. Read it as directional, not decisive.
• Data: /data/jobs/c3_h1/ (results.txt, h1_clean.log, adapters 1–2).

H2–H6: not yet run — GPU-gated, and paused on the pre-launch fixes: a proper 3-seed protocol, robust judge belief-extraction (the P=X read is fragile), and a faithful-D re-distill.